# Self-training with Noisy Student improves ImageNet classification

**Authors:** Qizhe Xie, Eduard Hovy, Minh-Thang Luong, Quoc V. Le  
**arXiv:** [1911.04252](https://arxiv.org/abs/1911.04252)  
**Date:** 2019-11-11  
**Summary page:** https://www.summarizepaper.com/en/paper/1911.04252v1/

## Key points

- Authors: Qizhe Xie, Eduard Hovy, Minh-Thang Luong, Quoc V. Le
- Self-training method significantly enhances ImageNet classification accuracy
- Achieves 87.4% top-1 accuracy, surpassing state-of-the-art model by 1.0%
- ImageNet-A top-1 accuracy increased from 16.6% to 74.2%
- Mean corruption error on ImageNet-C reduced from 45.7 to 31.2
- Mean flip rate on ImageNet-P decreased from 27.8 to 16.1
- Methodology involves training an EfficientNet teacher model initially on labeled images and using it to generate pseudo labels for unlabeled images
- Larger EfficientNet student model is trained using both labeled and pseudo-labeled data in combination
- Iterative process continues with student model becoming the new teacher in subsequent rounds of training
- No noise introduced during pseudo label generation; noise deliberately injected during student learning phase for improved performance based on noisy labels

## Summary

In their paper titled "Self-training with Noisy Student improves ImageNet classification," authors Qizhe Xie, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le introduce a simple self-training method that significantly enhances image classification accuracy on the ImageNet dataset. The proposed approach achieves an impressive 87.4% top-1 accuracy, surpassing the state-of-the-art model by 1.0%, which relies on a massive dataset of 3.5 billion weakly labeled Instagram images. Notably, the self-training method demonstrates remarkable improvements on robustness test sets as well. It elevates the ImageNet-A top-1 accuracy from 16.6% to an impressive 74.2%. Furthermore, it effectively reduces the mean corruption error on ImageNet-C from 45.7 to 31.2 and decreases the mean flip rate on ImageNet-P from 27.8 to 16.1. The methodology involves training an EfficientNet model initially on labeled ImageNet images, using it as a teacher to generate pseudo labels for a vast set of 300 million unlabeled images. Subsequently, a larger EfficientNet model is trained as a student using both labeled and pseudo-labeled data in combination. This iterative process continues by utilizing the student model as the new teacher in subsequent rounds of training. Notably, during the generation of pseudo labels, no noise is introduced to ensure high-quality labels are produced by the teacher model. However, during the learning phase of the student model, various forms of noise such as data augmentation, dropout techniques, and stochastic depth are injected deliberately to challenge and enhance the learning process for improved performance based on noisy labels. Overall, this innovative self-training approach with a noisy student not only achieves superior performance on ImageNet classification but also showcases significant advancements in robustness testing scenarios compared to existing state-of-the-art models in image recognition tasks.