# Training Instabilities Induce Flatness Bias in Gradient Descent

**Authors:** Lawrence Wang, Stephen J. Roberts  
**arXiv:** [2511.12558](https://arxiv.org/abs/2511.12558)  
**Date:** 2025-11-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2511.12558v1/

## Key points

- Lawrence Wang and Stephen J. Roberts explore training instabilities in deep learning
- Training instabilities induce a flatness bias in gradient descent optimization algorithms
- Instabilities drive parameters towards flatter regions of the loss landscape, improving generalization capabilities
- Introduction of Rotational Polarity of Eigenvectors (RPE) concept where leading eigenvectors of the Hessian rotate during training instabilities
- Higher learning rates lead to more pronounced rotations, facilitating exploration and resulting in flatter minima
- Theoretical framework extends to stochastic gradient descent, showing instability-driven flattening persists even with minibatch noise
- Experimentation with restoring instabilities in Adam optimizer leads to further improvements in generalization performance

## Summary

In their paper titled "Training Instabilities Induce Flatness Bias in Gradient Descent," Lawrence Wang and Stephen J. Roberts delve into the phenomenon of training instabilities in deep learning. They specifically focus on the implicit bias it induces in gradient descent (GD) optimization algorithms. Classical analyses of GD typically define a stability threshold based on the largest eigenvalue of the loss Hessian, known as sharpness. This is where training is considered stable and the loss decreases monotonically. However, modern deep networks often exhibit superior performance beyond this stability threshold. Wang and Roberts demonstrate that these instabilities actually play a constructive role by driving parameters towards flatter regions of the loss landscape. This ultimately leads to improved generalization capabilities for deep learning models. The authors introduce the concept of Rotational Polarity of Eigenvectors (RPE), which is a geometric phenomenon where the leading eigenvectors of the Hessian rotate during training instabilities. These rotations become more pronounced with higher learning rates and facilitate exploration, resulting in flatter minima. Furthermore, their theoretical framework extends to stochastic GD, showing that instability-driven flattening persists even in the presence of minibatch noise. This has significant empirical effects and further supports their findings. The authors also experiment with restoring instabilities in Adam optimizer and observe further improvements in generalization performance. Overall, Wang and Roberts' research challenges traditional notions of stability thresholds and highlights how embracing training instabilities can lead to enhanced model generalization capabilities.