# KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization

**Authors:** Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, Anshumali Shrivastava  
**arXiv:** [2405.03917](https://arxiv.org/abs/2405.03917)  
**Date:** 2024-05-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.03917v1/

## Key points

- Efficient deployment of Large Language Models (LLMs) requires batching multiple requests together to improve throughput.
- Increasing batch size, context length, or model size can lead to the key and value (KV) cache size becoming a significant contributor to GPU memory usage and inference latency bottleneck.
- Coupled Quantization (CQ) is a new approach proposed to address this issue by exploiting high inter-dependency among distinct channels of key/value activation embeddings.
- CQ outperforms existing methods in preserving model quality even at extremely low bit widths through efficient encoding of activations by coupling multiple key/value channels together.
- By dividing key and value embedding channels into non-overlapping groups and estimating joint entropy and sum of marginal entropies for each group, CQ achieves efficient compression while maintaining encoding quality.
- The total amount of information needed for encoding decreases as the number of jointly quantized channels increases due to the slower growth rate of joint entropy compared to marginal entropies, highlighting the effectiveness of coupling multiple key/value channels in improving information efficiency during inference.

## Summary

Efficient deployment of Large Language Models (LLMs) requires batching multiple requests together to improve throughput. However, as the batch size, context length, or model size increases, the key and value (KV) cache size can quickly become a significant contributor to GPU memory usage and inference latency bottleneck. To address this issue, a new approach called Coupled Quantization (CQ) is proposed based on the observation that distinct channels of a key/value activation embedding exhibit high inter-dependency. This allows for more efficient encoding of activations by coupling multiple key/value channels together. Extensive experiments demonstrate that CQ outperforms existing methods in preserving model quality even at extremely low bit widths. By dividing the channels of key and value embeddings into non-overlapping groups and estimating joint entropy and sum of marginal entropies for each group, CQ achieves efficient compression while maintaining encoding quality. The analysis shows that as the number of jointly quantized channels increases, the total amount of information needed for encoding decreases due to the slower growth rate of joint entropy compared to marginal entropies. This highlights the effectiveness of coupling multiple key/value channels in improving information efficiency during inference.