# Model Compression and Efficient Inference for Large Language Models: A Survey

**Authors:** Wenxiao Wang, Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, Xiaofei He  
**arXiv:** [2402.09748](https://arxiv.org/abs/2402.09748)  
**Date:** 2024-02-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.09748v1/

## Key points

- The study focuses on challenges and solutions for deploying large language models on resource-constrained devices
- Compression and acceleration algorithms for large language models are categorized into quantization, pruning, distillation, compact architecture design, and dynamic networks
- Differentiating characteristics of large models include the need for fine-tuning or retraining post-compression and prioritizing versatility over performance on a single task
- Medium-sized models are distinguished from "real" large models based on these characteristics
- Mature frameworks for efficient inference of large models are introduced to support basic compression or acceleration algorithms

## Summary

In their paper titled "Model Compression and Efficient Inference for Large Language Models: A Survey," authors Wenxiao Wang, Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofei He delve into the challenges and solutions surrounding the deployment of large language models on resource-constrained devices. The study focuses on exploring compression and efficient inference methods for these models from an algorithmic perspective. Transformer-based large language models have seen remarkable success but face obstacles due to high memory and computational costs during inference. The authors categorize compression and acceleration algorithms for large language models into quantization, pruning, distillation, compact architecture design, and dynamic networks. They highlight two key characteristics that differentiate large models from smaller ones. Firstly, most compression algorithms for large models require fine-tuning or even retraining post-compression, which incurs a significant cost. To address this issue, researchers are exploring tuning-free algorithms like quantization and pruning. Secondly, large models prioritize versatility and generalization over performance on a single task. Algorithms such as knowledge distillation aim to preserve these qualities post-compression. The paper distinguishes between medium-sized models and "real" large models based on these characteristics. Additionally, the authors introduce mature frameworks for efficient inference of large models that support basic compression or acceleration algorithms. This introduction aims to facilitate model deployment for users by providing them with the necessary tools. Overall, the study spans 47 pages and reviews 380 papers as part of ongoing work in the field of model compression and efficient inference for large language models. Through their research efforts, the authors shed light on innovative approaches to address the challenges associated with deploying large language models efficiently on various devices while maintaining their effectiveness and versatility in different tasks.