# Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates

**Authors:** Leslie N. Smith, Nicholay Topin  
**arXiv:** [1708.07120](https://arxiv.org/abs/1708.07120)  
**Date:** 2017-08-23  
**Summary page:** https://www.summarizepaper.com/en/paper/1708.07120v3/

## Key points

- Super-convergence revolutionizes neural network training, leading to significantly faster training times compared to traditional methods.
- Key component: Single learning rate cycle and large maximum learning rate used during training.
- Role in regularization: Requires reduction in other forms of regularization for optimal balance.
- Introduction of simplified Hessian Free optimization method to estimate optimal learning rate.
- Effectiveness demonstrated through experiments on various datasets and architectures like resnet, wide-resnet, densenet, and inception.
- Performance improvements observed compared to standard training methods when labeled data is limited.
- Code and architectures available on github.com/lnsmith54/super-convergence for replication purposes.
- Application in winning the DAWNBench challenge showcases practical relevance in real-world scenarios.

## Summary

The discovery of super-convergence has revolutionized the training of neural networks, allowing for significantly faster training times compared to traditional methods. This phenomenon is crucial in understanding why deep networks are able to effectively generalize. One key component of super-convergence is the use of a single learning rate cycle and a large maximum learning rate during training. The authors emphasize that this approach also plays a role in regularization, requiring a reduction in other forms of regularization for optimal balance. They also introduce a simplified version of the Hessian Free optimization method to estimate the optimal learning rate. Through experiments on various datasets and architectures such as resnet, wide-resnet, densenet, and inception, the authors demonstrate the effectiveness of super-convergence. They show that it leads to significant performance improvements compared to standard training methods when labeled data is limited. Additionally, they provide access to their code and architectures used in their experiments for replication purposes on github.com/lnsmith54/super-convergence. Furthermore, the authors showcase an application of super-convergence in winning the DAWNBench challenge, highlighting its practical relevance in real-world scenarios. Overall, this paper presents super-convergence as a general fast training methodology with implications for enhancing the efficiency and effectiveness of neural network training processes.