# Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application

**Authors:** Chuanpeng Yang, Wang Lu, Yao Zhu, Yidong Wang, Qian Chen, Chenlong Gao, Bingjie Yan, Yiqiang Chen  
**arXiv:** [2407.01885](https://arxiv.org/abs/2407.01885)  
**Date:** 2024-07-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.01885v1/

## Key points

- Large Language Models (LLMs) pose challenges in practical deployment due to their substantial size and computational demands.
- Knowledge distillation (KD) is an effective technique for compressing LLMs while maintaining accuracy and enhancing inference speed.
- The paper by Yang et al. categorizes KD methods into white-box KD and black-box KD, highlighting their differences and exploring evaluation tasks and distillation effects.
- The authors provide valuable insights into the latest advancements and practical applications of KD for LLMs, paving the way for sustained progress in this field.

## Summary

In their paper titled "Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application," authors Chuanpeng Yang, Wang Lu, Yao Zhu, Yidong Wang, Qian Chen, Chenlong Gao, Bingjie Yan, and Yiqiang Chen delve into the realm of Large Language Models (LLMs) and the challenges they pose in practical deployment due to their substantial size and computational demands. Despite the impressive capabilities of LLMs across various domains, the need to compress these models while maintaining accuracy has become a focal point of research. Among the methods explored in this survey, knowledge distillation emerges as an effective technique to enhance inference speed without significantly compromising performance. (LLMs) have revolutionized natural language processing with their impressive capabilities. However, (KD) has emerged as a crucial technique for optimizing LLM performance in resource-constrained environments. In their paper titled "Survey on KD for LLMs," Yang et al. provide a comprehensive overview from three key aspects: method,, and . They categorize KD methods into white-box KD and black-box KD to highlight their differences and explore evaluation tasks and distillation effects across different methods. The authors' exploration of  offers valuable insights into the latest advancements and practical applications in this field. By providing a deeper understanding of how KD can be leveraged effectively to optimize LLM performance, The findings presented pave the way for sustained progress by proposing directions for future research in this domain.