# InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU

**Authors:** Heejun Lee, Geon Park, Jaduk Suh, Sung Ju Hwang  
**arXiv:** [2502.08910](https://arxiv.org/abs/2502.08910)  
**Date:** 2025-02-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2502.08910v1/

## Key points

- Challenge of handling very long context lengths in modern large language models (LLMs)
- Introduction of a novel LLM inference framework to address slow inference speeds and increased memory costs
- Framework accelerates processing by dynamically eliminating irrelevant context tokens through hierarchical token pruning algorithm
- Allows for generalization to longer sequences by applying RoPE adjustment methods based on internal attention patterns within LLMs
- Offloading key-value cache to host memory during inference results in substantial reduction in GPU memory pressure
- Enables processing of up to 3 million tokens on a single L40s 48GB GPU without permanent loss of context information
- Achieves an impressive 18.95x speedup in attention decoding for a 1 million token context without additional training
- Implemented within the SGLang framework, demonstrating effectiveness and practicality through evaluations on LongBench and ��Bench benchmarks
- Superior performance and practicality showcased in latency benchmarks compared to previous state-of-the-art approaches
- Potential to enhance energy efficiency and reduce inference latency without altering trained behavior of existing Transformer models

## Summary

In the realm of modern large language models (LLMs), the challenge of handling very long context lengths has been a significant hurdle. This has led to slower inference speeds and increased memory costs. To address these issues and enable efficient utilization of long contexts, the  framework is introduced. This novel LLM inference framework accelerates processing by dynamically eliminating irrelevant context tokens through a modular hierarchical token pruning algorithm. Additionally, it allows for generalization to longer sequences by selectively applying various RoPE adjustment methods based on internal attention patterns within LLMs. One key feature of  is the offloading of key-value cache to host memory during inference. This results in a substantial reduction in GPU memory pressure. As a result,  enables the processing of up to 3 million tokens on a single L40s 48GB GPU – three times larger than previous capabilities – without any permanent loss of context information. The framework achieves an impressive 18.95x speedup in attention decoding for a 1 million token context without requiring additional training. Implemented within the SGLang framework,  demonstrates its effectiveness and practicality through extensive evaluations on LongBench and ��Bench benchmarks. The latency benchmarks showcase the superior performance and practicality of this method over previous state-of-the-art approaches. Looking ahead, we believe that  has the potential to significantly enhance energy efficiency and reduce inference latency without altering the trained behavior of existing Transformer models. With strong results in performance recovery and faster processing speeds, this method is poised to offer substantial benefits for production use in the future.