# EdgeInfinite: A Memory-Efficient Infinite-Context Transformer for Edge Devices

**Authors:** Jiyu Chen, Shuang Peng, Daxiong Luo, Fan Yang, Renshou Wu, Fangyuan Li, Xiaoxin Chen  
**arXiv:** [2503.22196](https://arxiv.org/abs/2503.22196)  
**Date:** 2025-03-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2503.22196v1/

## Key points

- EdgeInfinite is a groundbreaking solution designed to address challenges faced by Transformer-based large language models (LLMs) processing long sequences on edge devices.
- Challenges stem from quadratic complexity of attention mechanisms and increasing memory demands from Key-Value (KV) cache.
- Existing KV cache optimizations struggle with irreversible token eviction in tasks requiring long outputs.
- EdgeInfinite introduces innovative solution for handling infinite contexts within Transformer-based LLMs, incorporating compressed memory into models through a .
- Maintains full compatibility with standard Transformer architectures and requires minimal parameter fine-tuning.
- Enables selective activation of for routing tasks involving both long and short contexts.
- Experimental results show EdgeInfinite achieves performance levels comparable to baseline Transformer-based LLMs on benchmarks with long contexts, optimizing memory consumption and reducing processing time for the first token.

## Summary

EdgeInfinite is a groundbreaking solution designed to address the challenges faced by Transformer-based large language models (LLMs) when processing long sequences on edge devices. These challenges stem from the quadratic complexity of attention mechanisms and the increasing memory demands from Key-Value (KV) cache. Existing KV cache optimizations struggle with irreversible token eviction in tasks that require long outputs. Additionally, alternative sequence modeling architectures are costly to integrate into established Transformer infrastructure. In response to these issues, EdgeInfinite introduces a  for handling infinite contexts within Transformer-based LLMs. This innovative solution incorporates compressed memory into the models through a . Importantly, EdgeInfinite maintains full compatibility with standard Transformer architectures and requires only minimal parameter fine-tuning. This enables selective activation of the  for routing tasks that involve both long and short contexts. The experimental results demonstrate that EdgeInfinite achieves performance levels comparable to baseline Transformer-based LLMs on benchmarks involving long contexts. Notably, EdgeInfinite optimizes memory consumption and reduces the time required to process the first token. The authors of this study – Jiyu Chen, Shuang Peng, Daxiong Luo, Fan Yang, Renshou Wu, Fangyuan Li, and Xiaoxin Chen – have made a significant contribution to advancing the capabilities of edge devices in handling complex language processing tasks efficiently and effectively. Their work on  has paved the way for improved performance and efficiency in , providing a valuable solution for addressing challenges related to processing long sequences on edge devices.