# Adam: A Method for Stochastic Optimization

**Authors:** Diederik Kingma, Jimmy Ba  
**arXiv:** [1412.6980](https://arxiv.org/abs/1412.6980)  
**Date:** 2014-12-22  
**Summary page:** https://www.summarizepaper.com/en/paper/1412.6980v1/

## Key points

- Adam is an algorithm designed for first-order gradient-based optimization of stochastic objective functions.
- It relies on adaptive estimates of lower-order moments of the gradients.
- Adam is computationally efficient and has minimal memory requirements, making it suitable for problems with large amounts of data and/or parameters.
- It can handle non-stationary objectives and problems with noisy or sparse gradients by exhibiting invariance to diagonal rescaling of the gradients.
- The algorithm's hyperparameters have intuitive interpretations and typically require little tuning.
- The authors analyze its theoretical convergence properties and provide a regret bound on the convergence rate, comparable to the best-known results within the online convex optimization framework.
- Experimental comparisons demonstrate that Adam performs well in practice.
- Overall, Adam presents a reliable and efficient approach to optimizing stochastic objective functions using first-order gradients.

## Summary

Adam is an algorithm designed for first-order gradient-based optimization of stochastic objective functions. It offers a straightforward implementation and relies on adaptive estimates of lower-order moments of the gradients. This method is particularly suitable for problems that involve large amounts of data and/or parameters, as it is computationally efficient and has minimal memory requirements. One notable feature of Adam is its ability to handle non-stationary objectives and problems with noisy or sparse gradients. It achieves this by exhibiting invariance to diagonal rescaling of the gradients, adapting to the geometry of the objective function. The algorithm's hyperparameters have intuitive interpretations and typically require little tuning. Additionally, the authors analyze its theoretical convergence properties and provide a regret bound on the convergence rate. This bound is comparable to the best-known results within the online convex optimization framework. To validate its effectiveness, Adam was experimentally compared to other stochastic optimization methods; results demonstrate that Adam performs well in practice. Overall, Adam presents a reliable and efficient approach to optimizing stochastic objective functions using first-order gradients. Its adaptability to various problem characteristics makes it a valuable tool for researchers and practitioners working with large datasets or complex models.