# Landmark Attention: Random-Access Infinite Context Length for Transformers

**Authors:** Amirkeivan Mohtashami, Martin Jaggi  
**arXiv:** [2305.16300](https://arxiv.org/abs/2305.16300)  
**Date:** 2023-05-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.16300v1/

## Key points

- Transformers have limitations in handling longer contexts due to large memory requirements
- Previous approaches compromised random-access flexibility or relied on separate mechanisms for context retrieval
- The authors propose a novel approach using landmark tokens to represent input blocks and training the attention mechanism to select relevant blocks
- This eliminates the need for a separate mechanism and allows retrieval of blocks directly through the attention mechanism
- The method seamlessly integrates with specialized data structures and memory hierarchy for processing arbitrarily long context lengths
- Comparable performance with Transformer-XL is achieved while reducing the number of retrieved tokens in each step
- Fine-tuning LLaMA 7B with this method extends its context length capacity up to 32k tokens, allowing for inference at GPT-4's context lengths
- "Landmark Attention" addresses memory limitations of transformers when handling longer contexts
- Random-access flexibility is maintained while efficiently retrieving relevant blocks through the attention mechanism itself
- Experimental results demonstrate comparable performance with existing models and reduced computational requirements.

## Summary

The paper "Landmark Attention: Random-Access Infinite Context Length for Transformers" addresses the limitation of transformers in handling longer contexts due to their large memory requirements. Previous approaches compromised the random-access flexibility of attention or relied on separate mechanisms for context retrieval, which may not be compatible with the model's attention. To overcome these limitations, the authors propose a novel approach that allows access to the complete context while retaining random-access flexibility. Their method involves using a landmark token to represent each block of the input and training the attention mechanism to select relevant blocks using this landmark token. This enables retrieval of blocks directly through the attention mechanism, eliminating the need for a separate mechanism. The approach seamlessly integrates with specialized data structures and memory hierarchy, enabling processing of arbitrarily long context lengths. The authors demonstrate that their method achieves comparable performance with Transformer-XL while significantly reducing the number of retrieved tokens in each step. Additionally, they show that fine-tuning LLaMA 7B with their method extends its context length capacity up to 32k tokens, allowing for inference at GPT-4's context lengths. In summary, "Landmark Attention" presents an innovative solution to address the memory limitations of transformers when handling longer contexts. Their approach enables random-access flexibility while efficiently retrieving relevant blocks through the attention mechanism itself. The experimental results highlight its effectiveness in achieving comparable performance with existing models while significantly reducing computational requirements.