# Practical tradeoffs between memory, compute, and performance in learned optimizers

**Authors:** Luke Metz, C. Daniel Freeman, James Harrison, Niru Maheswaranathan, Jascha Sohl-Dickstein  
**arXiv:** [2203.11860](https://arxiv.org/abs/2203.11860)  
**Date:** 2022-03-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2203.11860v1/

## Key points

- The paper explores the role of optimization in developing machine learning systems
- It focuses on learned optimizers, which replace hand-designed optimizers with flexible parametric functions
- Learned optimizers have the potential to reduce training steps and improve test loss but can be computationally and memory costly
- The paper aims to identify design features that impact trade-offs between memory, compute, and performance for both learned and hand-designed optimizers
- An analysis is conducted to understand how different design choices affect these trade-offs
- A new learned optimizer is developed based on the analysis, which is faster and more memory efficient compared to previous approaches
- Optimizing parameters based on findings leads to improved performance while minimizing resource requirements
- The research provides insights into practical considerations for designing and using learned optimizers in machine learning systems

## Summary

The paper titled "Practical tradeoffs between memory, compute, and performance in learned optimizers" explores the role of optimization in developing machine learning systems. It focuses on learned optimizers, which replace the hyperparameters of commonly used hand-designed optimizers with flexible parametric functions that are optimized to minimize a target loss on a specific class of models. Learned optimizers have the potential to reduce the number of training steps required and improve the final test loss but can be costly due to computational and memory overhead. This paper aims to identify and quantify design features that impact the trade-offs between memory, compute, and performance for both learned and hand-designed optimizers. The authors conduct an analysis to understand how different design choices affect these trade-offs and leverage this analysis to develop a new learned optimizer that is faster and more memory efficient compared to previous approaches. By optimizing their parameters based on their findings they achieve improved performance while minimizing resource requirements. This research contributes valuable insights into the practical considerations involved in designing and using learned optimizers in machine learning systems which can help researchers and practitioners make informed decisions about optimizing models while balancing computational resources and performance goals.