# Recurrent Memory Transformer

**Authors:** Aydar Bulatov, Yuri Kuratov, Mikhail S. Burtsev  
**arXiv:** [2207.06881](https://arxiv.org/abs/2207.06881)  
**Date:** 2022-07-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2207.06881v1/

## Key points

- Authors explore challenges faced by Transformer-based models in handling global and local information within sequences
- Existing models show success in creating context-aware representations through self-attention mechanisms
- Storing both global and local information in element-wise representations presents limitations
- Quadratic computational complexity of self-attention restricts effective processing of longer input sequences
- Proposed solution: Recurrent Memory Transformer utilizes memory to store and process both local and global information, enabling information exchange between segments through recurrence
- Integration of memory mechanism achieved by introducing special memory tokens to input or output sequence
- Experimental results show Recurrent Memory Transformer performs comparably to Transformer-XL with smaller memory sizes, outperforms it for processing longer sequences effectively
- Inclusion of memory tokens enhances performance, making it suitable for tasks requiring learning long-term dependencies and versatile memory processing capabilities

## Summary

In their paper titled "Recurrent Memory Transformer," authors Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev explore the challenges faced by Transformer-based models in handling global and local information within sequences. These models have shown success in creating context-aware representations through self-attention mechanisms that combine information from all sequence elements. However, storing both global and local information in element-wise representations presents limitations. Additionally, the quadratic computational complexity of self-attention restricts the effective processing of longer input sequences. To address these challenges, the authors propose a novel approach - a memory-augmented segment-level recurrent Transformer known as the Recurrent Memory Transformer. This innovative model utilizes memory to store and process both local and global information while facilitating information exchange between segments through recurrence. The integration of this memory mechanism into the existing Transformer model is achieved by introducing special memory tokens to either the input or output sequence. Through training, the transformer learns to effectively manage both memory operations and sequence representation processing. Experimental results presented in the study demonstrate that the Recurrent Memory Transformer performs comparably to the established Transformer-XL model in language modeling tasks with smaller memory sizes. However, it outperforms Transformer-XL when tasked with processing longer sequences effectively. The inclusion of memory tokens further enhances its performance, highlighting its potential for applications requiring learning long-term dependencies and versatile memory processing capabilities such as algorithmic tasks and reasoning. Overall, this research introduces a promising architecture that addresses key limitations of existing Transformer models by incorporating a memory-augmented approach. This paves the way for enhanced performance in handling complex sequential data across various domains and tasks.