# Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

**Authors:** Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister  
**arXiv:** [2305.02301](https://arxiv.org/abs/2305.02301)  
**Date:** 2023-05-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.02301v1/

## Key points

- Large language models (LLMs) are memory inefficient and compute-intensive for practical applications
- Smaller task-specific models have been trained using finetuning or distillation, but both methods require large amounts of training data to achieve comparable performance to LLMs
- A new mechanism called Distilling step-by-step has been introduced that trains smaller models that outperform LLMs while leveraging less training data than finetuning or distillation
- The method extracts LLM rationales as additional supervision for small models within a multi-task training framework
- Experiments across four NLP benchmarks showed that the Distilling step-by-step mechanism achieved better performance with much fewer labeled/unlabeled training examples compared to both finetuning and distillation
- The method achieved better performance using substantially smaller model sizes compared to LLMs
- The study reduced both the model size and the amount of data required to outperform LLMs; their 770M T5 model outperformed the 540B PaLM model using only 80% of available data on a benchmark task.
- Authors of this study include Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee and Tomas Pfister.

## Summary

The deployment of large language models (LLMs) presents challenges due to their memory inefficiency and compute-intensive nature for practical applications. To address this, researchers have trained smaller task-specific models using either finetuning with human labels or distillation using LLM-generated labels. However, both methods require large amounts of training data to achieve comparable performance to LLMs. In response, a new mechanism called Distilling step-by-step has been introduced that trains smaller models that outperform LLMs while leveraging less training data than finetuning or distillation. The method extracts LLM rationales as additional supervision for small models within a multi-task training framework. The study conducted experiments across four NLP benchmarks and presented three key findings. Firstly, compared to both finetuning and distillation, the Distilling step-by-step mechanism achieved better performance with much fewer labeled/unlabeled training examples. Secondly, compared to LLMs, the method achieved better performance using substantially smaller model sizes. Thirdly, the study reduced both the model size and the amount of data required to outperform LLMs; their 770M T5 model outperformed the 540B PaLM model using only 80% of available data on a benchmark task. The authors of this study include Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee and Tomas Pfister. Their research paper titled "Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes" was accepted for publication in Findings of ACL 2023.