# Towards the Law of Capacity Gap in Distilling Language Models

**Authors:** Chen Zhang, Dawei Song, Zheyu Ye, Yan Gao  
**arXiv:** [2311.07052](https://arxiv.org/abs/2311.07052)  
**Date:** 2023-11-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.07052v1/

## Key points

- Language model (LM) distillation aims to transfer knowledge from a large teacher LM to a smaller student LM
- The curse of capacity gap makes distilling LMs difficult, suggesting that simply using a larger teacher LM does not always result in a better student LM
- There is an optimal point along the scaling course of the teacher LM that yields the best student LM
- Larger teacher LMs generally outperform smaller ones but require more resources
- The concept of the law of capacity gap reveals that the optimal capacity gap remains consistent across different scales and architectures of student LMs
- MiniMA, a 3B student LM distilled from a 7B teacher LM called adapted LLaMA2-7B, achieves new compute-performance pareto frontier among existing 3B LMs on benchmarks
- MiniChat, an instruction-tuned version of MiniMA, outperforms numerous 3B competitors in GPT4 evaluation and competes with several 7B chat models
- Effective distillation requires considering an optimal capacity gap between teacher and student LMs
- The findings offer insights for improving language model distillation techniques and achieving high performance with resource-efficient models.

## Summary

In the field of language model (LM) distillation, the goal is to transfer the knowledge from a large teacher LM to a smaller student LM. However, this process becomes challenging when there is a significant capacity gap between the two models. Despite various methods proposed to address this issue, distilling LMs remains difficult due to the curse of capacity gap. This curse suggests that simply using a larger teacher LM does not always result in a better student LM compared to using a smaller teacher LM. In fact, there seems to be an optimal point along the scaling course of the teacher LM that yields the best student LM. Previous studies have shown that this curse can only be partially lifted. On the other hand, while larger teacher LMs generally outperform smaller ones, they require more resources, especially in the context of recent large LMs (LLMs). Instead of trying to overcome the curse of capacity gap, it may be acceptable to leave it as it is. Interestingly, this paper introduces the concept of the law of capacity gap and reveals that the optimal capacity gap remains consistent across different scales and architectures of student LMs. This discovery allows for distilling a 3B student LM called MiniMA from a 7B teacher LM known as adapted LLaMA2-7B. The authors demonstrate that MiniMA achieves a new compute-performance pareto frontier among existing 3B LMs on commonly used benchmarks. Additionally, they introduce an instruction-tuned version called MiniChat which outperforms numerous 3B competitors in GPT4 evaluation and even competes with several 7B chat models. Overall, this research sheds light on how to effectively distill knowledge from large teacher LMs to smaller student LMs by considering an optimal capacity gap. The findings presented in this paper offer valuable insights for improving language model distillation techniques and achieving high performance with resource-efficient models.