# When Does Label Smoothing Help?

**Authors:** Rafael Müller, Simon Kornblith, Geoffrey Hinton  
**arXiv:** [1906.02629](https://arxiv.org/abs/1906.02629)  
**Date:** 2019-06-06  
**Summary page:** https://www.summarizepaper.com/en/paper/1906.02629v1/

## Key points

- Label smoothing involves using soft targets that are a weighted average of hard targets and a uniform distribution over labels to prevent the network from becoming over-confident.
- Label smoothing enhances generalization and improves model calibration, leading to significant enhancements in beam-search algorithms.
- Training a teacher network with label smoothing makes knowledge distillation into a student network less effective.
- Label smoothing encourages training examples from the same class to form tight clusters in representation space, improving generalization and calibration but leading to a loss of information in logits regarding similarities between instances of different classes.
- The study highlights the potential benefits of label smoothing for improving generalization and model calibration while also revealing its limitations in knowledge distillation scenarios.

## Summary

In their paper titled "When Does Label Smoothing Help? ", authors Rafael Müller, Simon Kornblith, and Geoffrey Hinton explore the impact of label smoothing on multi-class neural networks. Label smoothing involves using soft targets that are a weighted average of hard targets and a uniform distribution over labels to prevent the network from becoming over-confident. The technique has been widely used in various models such as image classification, language translation, and speech recognition. However, its underlying mechanisms are still not fully understood. Through empirical analysis, the authors demonstrate that label smoothing not only enhances generalization but also improves model calibration. This leads to significant enhancements in beam-search algorithms. However, they also observe that when a teacher network is trained with label smoothing, knowledge distillation into a student network becomes less effective. To explain these findings, the authors visualize how label smoothing influences the representations learned by the penultimate layer of the network. Their results show that label smoothing encourages training examples from the same class to form tight clusters in representation space. While this clustering improves generalization and calibration of predictions, it also leads to a loss of information in logits regarding similarities between instances of different classes. This loss of information hinders knowledge distillation processes but does not negatively impact overall model performance. Overall, this study sheds light on the nuanced effects of label smoothing on neural networks and highlights its potential benefits for improving generalization and model calibration while also revealing its limitations in knowledge distillation scenarios.