# Distillation Scaling Laws

**Authors:** Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, Russ Webb  
**arXiv:** [2502.08606](https://arxiv.org/abs/2502.08606)  
**Date:** 2025-02-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2502.08606v1/

## Key points

- The paper "Distillation Scaling Laws" by Dan Busbridge et al. analyzes distillation scaling laws.
- Authors propose a method to estimate model performance based on compute budget distribution between student and teacher models.
- Compute optimal distillation recipes are developed for scenarios with existing teacher models or when training is needed.
- Distillation outperforms supervised pretraining until reaching a predictable compute threshold that scales with student size or when multiple students are involved.
- In cases where only one student requires distillation and the teacher model also needs training, supervised learning is recommended over distillation.
- Valuable insights from large-scale examination of distillation techniques enhance understanding and provide guidance for experimental design in related research.
- The paper offers a robust framework for optimizing distillation practices and maximizing student performance within specified computational constraints.

## Summary

The paper "Distillation Scaling Laws" by Dan Busbridge et al. presents a comprehensive analysis of distillation scaling laws. The authors introduce a novel approach for estimating the performance of distilled models based on a specified compute budget and its distribution between the student and teacher models. This methodology effectively mitigates risks associated with implementing distillation at scale. One key contribution is the development of compute optimal distillation recipes tailored for scenarios where a teacher model already exists or when one needs training. The findings suggest that distillation surpasses supervised pretraining until reaching a predictable compute threshold that scales with student size when multiple students are to be distilled or if a teacher model is already in place. However, in cases where only one student requires distillation and the teacher model also necessitates training, supervised learning is recommended over distillation. Furthermore, the study offers valuable insights gleaned from an extensive large-scale examination of distillation techniques. These insights not only enhance our understanding of distillation processes but also provide valuable guidance for experimental design in related research endeavors. With 67 pages, 54 figures, and 13 tables included in their analysis, the authors present a robust framework for optimizing distillation practices and maximizing student performance within specified computational constraints.