# ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition

**Authors:** Lu Ye, Ze Tao, Yong Huang, Yang Li  
**arXiv:** [2402.15220](https://arxiv.org/abs/2402.15220)  
**Date:** 2024-02-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.15220v1/

## Key points

- Prefix-aware self-attention module
- Optimizes compute and memory operation costs
- Detects shared system prompts in prefixes across multiple LLM requests
- Improves memory utilization of KV cache
- Enhances data locality during self-attention computation with shared system prompts
- Accelerates self-attention kernel by 3.2-4.8 times compared to existing implementations
- Effective for system prompt lengths ranging from 1024 to 4096 tokens
- Importance of placing system prompts at the beginning of input contexts for effective ChunkAttention utilization
- Incorporating domain knowledge efficiently without separate model instances for each application
- Potential benefits in multi-tenant LLM scenarios as hardware and software environments evolve

## Summary

The paper "ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition" by Lu Ye, Ze Tao, Yong Huang, and Yang Li addresses the challenge of inference latency in large language models (LLMs) caused by self-attention mechanisms for long sequences. The authors introduce ChunkAttention, a prefix-aware self-attention module that optimizes compute and memory operation costs by detecting shared system prompts in prefixes across multiple LLM requests. By breaking key/value tensors into smaller chunks and organizing them into an auxiliary prefix tree, ChunkAttention improves memory utilization of KV cache. Additionally, a two-phase partition algorithm enhances data locality during self-attention computation when shared system prompts are present. The study demonstrates that ChunkAttention accelerates the self-attention kernel by 3.2-4.8 times compared to existing implementations for system prompt lengths ranging from 1024 to 4096 tokens. The authors highlight the importance of placing system prompts at the beginning of input contexts to leverage ChunkAttention effectively. They also discuss the potential benefits of fine-tuning LLMs to incorporate domain knowledge efficiently without requiring separate model instances for each application. As hardware and software environments evolve, fine-tuning may become more practical and popular in multi-tenant LLM scenarios. Overall, ChunkAttention offers a promising solution to optimize self-attention mechanisms in large language models serving multiple applications while improving memory utilization and computational efficiency in the presence of shared system prompts.