# Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt

**Authors:** Zhaozhuo Xu, Zirui Liu, Beidi Chen, Yuxin Tang, Jue Wang, Kaixiong Zhou, Xia Hu, Anshumali Shrivastava  
**arXiv:** [2305.11186](https://arxiv.org/abs/2305.11186)  
**Date:** 2023-05-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.11186v1/

## Key points

- Novel approach proposed to optimize trade-off between efficiency and accuracy of large language models (LLMs) with billions of parameters
- Compression techniques used to reduce computational and memory requirements for LLM inference, but often result in reduced predictive precision
- Unique input format introduced for compressed LLMs that improves generation quality for specific queries
- Prompt learning paradigm introduced to enhance accuracy of compressed LLMs by cultivating an additive prompt over them
- Empirical results show strategic prompt utilization allows compressed LLMs to match or exceed accuracy of original models across different datasets, tasks, and compression levels
- Importance of judicious input editing emphasized for compressed large models
- Potential advancements suggested in scaling LLMs on common hardware configurations

## Summary

. This research proposes a novel approach to optimize the trade-off between efficiency and accuracy of large language models (LLMs) with billions of parameters. Compression techniques are employed to reduce computational and memory requirements for LLM inference, but often lead to reduced predictive precision. The authors introduce a unique input format for compressed LLMs that differs from the original models and discover that selecting prompts with precision can significantly improve the generation quality of compressed LLMs for specific queries. A prompt learning paradigm is introduced to enhance the accuracy of compressed LLMs by cultivating an additive prompt over them. The empirical results demonstrate that strategic prompt utilization allows compressed LLMs to match or even exceed the accuracy of the original models, while exhibiting transferability across different datasets, tasks, and compression levels. This study emphasizes the importance of judicious input editing for compressed large models and suggests potential advancements in scaling LLMs on common hardware configurations.