# Retentive Network: A Successor to Transformer for Large Language Models

**Authors:** Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei  
**arXiv:** [2307.08621](https://arxiv.org/abs/2307.08621)  
**Date:** 2023-07-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.08621v4/

## Key points

- Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang and Furu Wei propose a new architecture called Retentive Network (RetNet) for large language models
- RetNet aims to achieve training parallelism, low-cost inference and good performance simultaneously
- The researchers derive the connection between recurrence and attention
- RetNet introduces the retention mechanism for sequence modeling
- RetNet supports three computation paradigms: parallel representation, recurrent representation, and chunkwise recurrent representation
- Experimental results demonstrate favorable scaling results during training and efficient inference with RetNet
- RetNet is positioned as a strong successor to the Transformer model for large language models
- Code for RetNet is provided at https://aka.ms/retnet

## Summary

In their paper titled "Retentive Network: A Successor to Transformer for Large Language Models," Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang and Furu Wei propose a new architecture called Retentive Network (RetNet) as a foundation for large language models. The authors aim to achieve training parallelism, low-cost inference and good performance simultaneously. The researchers begin by theoretically deriving the connection between recurrence and attention. They then introduce the retention mechanism for sequence modeling in RetNet. This mechanism supports three computation paradigms: parallel representation; recurrent representation; and chunkwise recurrent representation. The parallel representation allows for training parallelism enabling efficient processing of data. The recurrent representation enables low-cost $O(1)$ inference which improves decoding throughput reduces latency and optimizes GPU memory usage without compromising performance. The chunkwise recurrent representation facilitates efficient modeling of long sequences with linear complexity; each chunk is encoded in parallel while being recurrently summarized. Experimental results on language modeling demonstrate that RetNet achieves favorable scaling results during training and exhibits low-cost deployment with efficient inference. These properties position RetNet as a strong successor to the Transformer model for large language models. To facilitate further research and implementation of their proposed architecture the authors provide code for RetNet at https://aka.ms/retnet .