# LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression

**Authors:** Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, Dongmei Zhang  
**arXiv:** [2403.12968](https://arxiv.org/abs/2403.12968)  
**Date:** 2024-03-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.12968v1/

## Key points

- Task-agnostic prompt compression for enhancing generalizability and efficiency
- Novel data distillation procedure to extract knowledge from Language Models (LLMs) for prompt compression
- Formulation of prompt compression as a token classification problem using a Transformer encoder
- Lower latency and explicit learning of compression objective with smaller models like XLM-RoBERTa-large and mBERT
- Extensive evaluation on various datasets showing significant performance improvements over strong baselines
- Robust generalization capabilities across different Language Models (LLMs)
- 3x-6x faster speed compared to existing methods, accelerating end-to-end latency by 1.6x-2.9x
- Impressive compression ratios ranging from 2x-5x

## Summary

This paper by Zhuoshi Pan et al. delves into the realm of task-agnostic prompt compression to enhance generalizability and efficiency. The focus lies on addressing the redundancy present in natural language through the compression of prompts. Existing methodologies typically involve removing tokens or lexical units based on their information entropy derived from a causal language model like LLaMa-7B. However, a key challenge arises as information entropy may not be the most optimal metric for prompt compression due to its reliance on unidirectional context and potential failure to capture all essential information required for effective compression. To combat these issues effectively, the authors propose a novel data distillation procedure aimed at extracting knowledge from a Language Model (LLM) to compress prompts without sacrificing crucial information. Additionally, they introduce an extractive text compression dataset to further enhance their approach. Prompt compression is formulated as a token classification problem to ensure the fidelity of the compressed prompt compared to the original one. A Transformer encoder serves as the foundational architecture utilized to capture all necessary information for prompt compression from full bidirectional context. The proposed methodology not only leads to lower latency but also explicitly learns the compression objective with smaller models such as XLM-RoBERTa-large and mBERT. Extensive evaluation conducted on both in-domain and out-of-domain datasets including MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH showcases significant performance improvements over strong baselines. The model demonstrates robust generalization capabilities across various Language Models (LLMs). Moreover, it proves to be 3x-6x faster than existing prompt compression methods while accelerating end-to-end latency by 1.6x-2.9x with impressive compression ratios ranging from 2x-5x. In conclusion, this research presents an innovative approach towards efficient and faithful task-agnostic prompt compression that not only enhances performance but also showcases remarkable speed improvements over existing methodologies.