# Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling

**Authors:** Sanchit Gandhi, Patrick von Platen, Alexander M. Rush  
**arXiv:** [2311.00430](https://arxiv.org/abs/2311.00430)  
**Date:** 2023-11-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.00430v1/

## Key points

- The paper addresses the challenge of deploying large pre-trained speech recognition models in low-latency or resource-constrained environments.
- The authors propose a method called pseudo-labelling to distill the Whisper model into a smaller variant called Distil-Whisper.
- Pseudo-labelling involves assembling a large-scale open source dataset and using a word error rate (WER) heuristic to select high-quality pseudo labels for training.
- Distil-Whisper is 5.8 times faster and has 51% fewer parameters compared to the original Whisper model.
- Distil-Whisper achieves a WER within 1% on out of distribution test data in a zero shot transfer setting.
- Distil-Whisper maintains the robustness of the Whisper model to difficult acoustic conditions while being less prone to hallucination errors on long form audio.
- Distil-Whisper can be paired with Whisper for speculative decoding, leading to a 2 times speed up with mathematically identical outputs as the original model.
- The authors have made their training code, inference code, and models publicly accessible to support further research in this domain.

## Summary

The paper titled "Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling" addresses the challenge of deploying large pre-trained speech recognition models in low-latency or resource-constrained environments. To overcome this challenge, the authors propose a method called pseudo-labelling which involves assembling a large-scale open source dataset to distill the Whisper model into a smaller variant called Distil-Whisper. The authors use a word error rate (WER) heuristic to select only the highest quality pseudo labels for training. The resulting distilled model, Distil-Whisper is 5.8 times faster and has 51% fewer parameters compared to the original Whisper model. Remarkably, Distil-Whisper achieves a WER within 1% on out of distribution test data in a zero shot transfer setting. One key advantage of Distil-Whisper is that it maintains the robustness of the Whisper model to difficult acoustic conditions while being less prone to hallucination errors on long form audio. Additionally, Distil-Whisper is designed to be paired with Whisper for speculative decoding leading to a 2 times speed up while guaranteeing mathematically identical outputs as the original model. To support further research in this domain, the authors have made their training code, inference code and models publicly accessible which enables other researchers to build upon their work and explore new possibilities in low latency and resource constrained speech recognition models. Overall this paper presents an effective approach for knowledge distillation that addresses the challenges associated with deploying large models in real world applications.