# KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

**Authors:** Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami  
**arXiv:** [2401.18079](https://arxiv.org/abs/2401.18079)  
**Date:** 2024-01-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.18079v6/

## Key points

- The paper introduces KVQuant for low precision KV cache quantization in large language models (LLMs)
- Innovative methods in KVQuant include Per-Channel Key Quantization, Pre-RoPE Key Quantization, Non-Uniform KV Cache Quantization, and Per-Vector Dense-and-Sparse Quantization
- Existing solutions struggle to accurately represent activations in sub-4-bit precision
- KVQuant achieves less than 0.1 perplexity degradation with 3-bit quantization on datasets like Wikitext-2 and C4
- Enables serving the LLaMA-7B model with a context length of up to 1 million on a single A100-80GB GPU and up to 10 million on an 8-GPU system
- Custom CUDA kernels were developed for KVQuant, resulting in approximately 1.7x speedups compared to baseline fp16 matrix-vector multiplications for the LLaMA-7B model

## Summary

The paper "KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization" by Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami presents a novel approach to low precision KV cache quantization for large language models (LLMs). The authors introduce <kw>KVQuant</kw>, which incorporates several innovative methods to facilitate low precision KV cache quantization. These methods include Per-Channel Key Quantization, Pre-RoPE Key Quantization, Non-Uniform KV Cache Quantization, and Per-Vector Dense-and-Sparse Quantization. As LLMs are increasingly used in applications requiring large context windows, the memory consumption during inference is dominated by KV cache activations. While quantization has shown promise in compressing these activations, existing solutions struggle to accurately represent activations in sub-4-bit precision. To address this issue, the authors propose <kw>KVQuant</kw>, which achieves less than 0.1 perplexity degradation with 3-bit quantization on datasets like Wikitext-2 and C4. This surpasses existing approaches and enables serving the LLaMA-7B model with a context length of up to 1 million on a single A100-80GB GPU and up to 10 million on an 8-GPU system. Furthermore, custom CUDA kernels were developed for <kw>KVQuant</kw>, resulting in approximately 1.7x speedups compared to baseline fp16 matrix-vector multiplications for the LLaMA-7B model. This work demonstrates significant advancements in low precision KV cache quantization for LLMs, paving the way for more efficient and scalable inference processes in large language models.