# Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache

**Authors:** Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wencong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, Shen Li, Zhigang Ji, Tao Xie, Yong Li, Wei Lin  
**arXiv:** [2401.02669](https://arxiv.org/abs/2401.02669)  
**Date:** 2024-01-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.02669v2/

## Key points

- Authors introduce Infinite-LLM as a novel approach to address challenges posed by dynamic nature of Large Language Models (LLMs) in service systems
- Autoregressive nature of LLMs leads to highly dynamic behavior in attention layers, causing notable differences in computational characteristics and memory requirements compared to non-attention layers
- Existing static model parallelism and resource allocation strategies struggle to cope with this dynamicity
- Infinite-LLM separates attention layers from an LLM's inference process, enabling flexible and independent resource scheduling for optimized computational performance and enhanced memory utilization
- Utilizes pooled GPU memory strategy across a cluster environment with 32 A100 GPUs to significantly improve system throughput and support extensive context lengths ranging from a few tokens to 2000K tokens
- Through evaluation on diverse context lengths dataset, Infinite-LLM demonstrates throughput enhancements of 1.35-3.4x compared to existing state-of-the-art methods
- Enables efficient and elastic deployment of LLMs by effectively managing dynamic context lengths and optimizing system performance through innovative resource scheduling techniques

## Summary

In their paper titled "Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache," authors Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wencong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, Shen Li, Zhigang Ji, Tao Xie,
Yong Li and Wei Lin introduce a novel approach to address the challenges posed by the dynamic nature of Large Language Models (LLMs) in service systems. <br>
 have shown significant potential in various domains through request serving. However,
as the demand for larger context sizes increases,
the autoregressive nature of LLMs leads to highly dynamic behavior in attention layers. This results in notable differences in computational characteristics and memory requirements compared to non-attention layers. These dynamics present significant obstacles for  and  in service systems. Existing static model parallelism and resource allocation strategies struggle to cope with this dynamicity. To tackle this issue effectively,
the authors propose Infinite-LLM as a solution designed to handle varying context lengths efficiently. Infinite-LLM separates attention layers from an LLM's inference process,
enabling flexible and independent resource scheduling that optimizes computational performance
and enhances memory utilization simultaneously. By utilizing a pooled GPU memory strategy across a cluster environment with 32 A100 GPUs,
Infinite-LLM not only significantly improves system throughput but also supports extensive context lengths ranging from a few tokens to 2000K tokens. Through evaluation on a dataset with diverse context lengths across the cluster setup mentioned above,
Infinite-LLM demonstrates throughput enhancements of 1.35-3.4x compared to existing state-of-the-art methods. Overall, Infinite-LLM enables efficient and elastic deployment of LLMs by effectively managing dynamic context lengths
and optimizing system performance through innovative resource scheduling techniques. The proposed approach showcases promising advancements in addressing the challenges associated with deploying large language models in real-world service systems.