# LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression

**Authors:** Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu  
**arXiv:** [2310.06839](https://arxiv.org/abs/2310.06839)  
**Date:** 2023-10-10  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2310.06839v1/

## Key points

- LongLLMLingua is a proposed solution for addressing challenges faced by large language models (LLMs) in long context scenarios.
- Challenges include higher computational/financial cost, longer latency, and inferior performance.
- LongLLMLingua aims to improve LLMs' perception of key information through prompt compression.
- Evaluation of LongLLMLingua covers various long context scenarios such as single-/multi-document QA, few-shot learning, summarization, synthetic tasks, and code completion.
- Experimental results show that LongLLMLingua's compressed prompts achieve higher performance with reduced cost.
- On the NaturalQuestions benchmark, LongLLMLingua achieves a performance boost of up to 17.1% compared to the original prompt while using approximately 4 times fewer tokens as input to GPT-3.5-Turbo.
- LongLLMLingua offers significant cost savings in benchmarks like LongBench and ZeroScrolls amounting to $28.5 and $27.4 per 1,000 samples respectively.
- When compressing prompts with a compression rate of 2x-10x (~10k tokens), LongLLMLingua speeds up the end-to-end latency by 1.4x-3.8x.
- Authors of this work are Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin Yuqing Yang and Lili Qiu who published their paper titled "LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression" providing further details on this approach for improving LLMs in long context scenarios.

## Summary

LongLLMLingua is a proposed solution to address the challenges faced by large language models (LLMs) in long context scenarios. These challenges include higher computational/financial cost, longer latency, and inferior performance. The performance of LLMs is influenced by the density and position of key information in the input prompt. LongLLMLingua aims to improve LLMs' perception of key information through prompt compression. The evaluation of LongLLMLingua covers various long context scenarios such as single-/multi-document QA, few-shot learning, summarization, synthetic tasks, and code completion. Experimental results demonstrate that LongLLMLingua's compressed prompts achieve higher performance with reduced cost. For instance, on the NaturalQuestions benchmark, LongLLMLingua achieves a performance boost of up to 17.1% compared to the original prompt while using approximately 4 times fewer tokens as input to GPT-3.5-Turbo. Furthermore, LongLLMLingua offers significant cost savings in benchmarks like LongBench and ZeroScrolls amounting to $28.5 and $27.4 per 1,000 samples respectively. Additionally, when compressing prompts with a compression rate of 2x-10x (~10k tokens), LongLLMLingua speeds up the end-to-end latency by 1.4x-3.8x. The authors of this work are Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin Yuqing Yang and Lili Qiu who published their paper titled "LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression" providing further details on this approach for improving LLMs in long context scenarios.