# Improving the convergence of SGD through adaptive batch sizes

**Authors:** Scott Sievert, Shrey Shah  
**arXiv:** [1910.08222](https://arxiv.org/abs/1910.08222)  
**Date:** 2019-10-18  
**Summary page:** https://www.summarizepaper.com/en/paper/1910.08222v4/

## Key points

- Research focuses on improving convergence of stochastic gradient descent (SGD) through adaptive batch sizes
- Challenges with small batch sizes leading to high-variance gradient estimates
- Proposal of a method that adapts batch size based on model's training loss for balanced efficiency and accuracy
- Adaptive approach requiring similar model updates as traditional gradient descent while maintaining comparable computational complexity to SGD
- Evaluation of model's loss on entire dataset after each update using an approximation technique for efficient estimation
- Experimental results showing fewer model updates without increased computation time with adaptive batch size strategy
- Discussion of related optimization approaches including step size decay schedules, ASGD, Adagrad, and Adam techniques for adjusting optimization based on informative features and gradients' moments.

## Summary

The research presented in this paper focuses on improving the convergence of stochastic gradient descent (SGD) through adaptive batch sizes. Mini-batch SGD and its variants are commonly used in optimization tasks, where the objective function's gradient is approximated using a small number of training examples known as the batch size. However, small batch sizes can lead to high-variance gradient estimates, posing challenges for optimization. On the other hand, large batches require more computation but yield higher precision gradient estimates. To address these challenges, the authors propose a method that adapts the batch size based on the model's training loss. By adjusting the batch size dynamically, the method aims to achieve a balance between computational efficiency and accurate gradient estimation. The study demonstrates that this adaptive approach requires a similar number of model updates as traditional gradient descent while maintaining a comparable level of computational complexity to SGD. One key aspect of this method is evaluating the model's loss on the entire dataset after each model update. To reduce computational overhead, an approximation technique is employed to estimate the training loss efficiently. Experimental results showcase that this adaptive batch size strategy leads to fewer model updates without increasing overall computation time. The paper also discusses related work in the field of optimization, highlighting approaches such as step size decay schedules and averaging model iterates with methods like ASGD. Additionally, adaptive learning rates techniques like Adagrad and Adam are explored for their ability to adjust optimization based on informative features and gradients' first and second moments. Overall, this research contributes valuable insights into optimizing SGD through adaptive batch sizes, offering a promising approach to enhance convergence rates in machine learning models while managing computational resources effectively.