# Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

**Authors:** Tsendsuren Munkhdalai, Manaal Faruqui, Siddharth Gopal  
**arXiv:** [2404.07143](https://arxiv.org/abs/2404.07143)  
**Date:** 2024-04-10  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2404.07143v2/

## Key points

- Paper introduces a novel method for scaling Transformer-based Large Language Models (LLMs) to handle infinitely long inputs
- Key innovation is the development of a new attention mechanism called Infini-attention
- Technique integrates compressive memory, masked local attention, and long-term linear attention mechanisms within a single Transformer block
- Demonstrated effectiveness through experiments on various language modeling benchmarks
- Showcased performance on tasks such as 1M sequence length passkey context block retrieval and 500K length book summarization using LLMs with 1B and 8B parameters
- Introduces minimal bounded memory parameters for fast streaming inference for LLMs
- Addresses the challenge of efficiently processing infinitely long inputs while maintaining computational constraints
- Infini-attention opens up new possibilities for handling complex language tasks without compromising performance or resource limitations

## Summary

The paper "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" by Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal introduces a novel method for scaling Transformer-based Large Language Models (LLMs) to handle infinitely long inputs while maintaining bounded memory and computation requirements. The key innovation in their approach is the development of a new attention mechanism called Infini-attention. This technique integrates compressive memory into the traditional attention mechanism and incorporates both masked local attention and long-term linear attention mechanisms within a single Transformer block. Through experiments on various language modeling benchmarks, the authors demonstrate the effectiveness of their proposed approach. They showcase its performance on tasks such as 1M sequence length passkey context block retrieval and 500K length book summarization using LLMs with 1B and 8B parameters. Importantly, their method introduces minimal bounded memory parameters, enabling fast streaming inference for LLMs. Overall, this work presents a significant advancement in natural language processing by addressing the challenge of efficiently processing infinitely long inputs while maintaining computational constraints. The incorporation of Infini-attention opens up new possibilities for handling complex language tasks that require extensive contextual information without compromising on performance or resource limitations.