Scaling Exponents Across Parameterizations and Optimizers

AI-generated keywords: Scaling Exponents Parameterizations Optimizers Hyperparameter Tuning Machine Learning Scalability

AI-generated Key Points

  • The paper explores scaling models from small to large width by adjusting algorithmic and architectural details such as parameterization and optimizer choices.
  • The authors challenge key assumptions in prior work regarding the alignment between parameters and data, proposing a new perspective on parameterization.
  • Through empirical investigations involving tens of thousands of models trained with various optimizers, parameterizations, alignment assumptions, learning rates, and model sizes up to 26.8B parameters, the authors derive new theoretical results under weaker assumptions.
  • Findings reveal that the best learning rate scaling prescription may have been overlooked in previous studies due to restrictive assumptions.
  • All parameterizations can achieve hyperparameter transfer contrary to existing beliefs that only maximal update parameterization (muP) can achieve this feat.
  • Introduction of a novel per-layer learning rate prescription for standard parameterization that outperforms muP is highlighted.
  • Importance of correctly scaling the epsilon parameter in Adam optimizer to prevent gradient underflow is emphasized.
  • Proposal of Adam-atan2 as a new numerically stable version of Adam that eliminates the epsilon hyperparameter entirely is made.
  • The study's comprehensive analysis sheds light on the intricate relationship between parameterization choices and optimizer performance across various model sizes, contributing valuable insights to machine learning scalability.
Also access our AI generated: Comprehensive summary, Lay summary, Blog-like article; or ask questions about this paper to our AI assistant.

Authors: Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A. Alemi, Roman Novak, Peter J. Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, Jeffrey Pennington

63 pages, International Conference on Machine Learning 2024
License: CC BY 4.0

Abstract: Robust and effective scaling of models from small to large width typically requires the precise adjustment of many algorithmic and architectural details, such as parameterization and optimizer choices. In this work, we propose a new perspective on parameterization by investigating a key assumption in prior work about the alignment between parameters and data and derive new theoretical results under weaker assumptions and a broader set of optimizers. Our extensive empirical investigation includes tens of thousands of models trained with all combinations of three optimizers, four parameterizations, several alignment assumptions, more than a dozen learning rates, and fourteen model sizes up to 26.8B parameters. We find that the best learning rate scaling prescription would often have been excluded by the assumptions in prior work. Our results show that all parameterizations, not just maximal update parameterization (muP), can achieve hyperparameter transfer; moreover, our novel per-layer learning rate prescription for standard parameterization outperforms muP. Finally, we demonstrate that an overlooked aspect of parameterization, the epsilon parameter in Adam, must be scaled correctly to avoid gradient underflow and propose Adam-atan2, a new numerically stable, scale-invariant version of Adam that eliminates the epsilon hyperparameter entirely.

Submitted to arXiv on 08 Jul. 2024

Ask questions about this paper to our AI assistant

You can also chat with multiple papers at once here.

AI assistant instructions?

Results of the summarizing process for the arXiv paper: 2407.05872v1

In their paper titled "Scaling Exponents Across Parameterizations and Optimizers," Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A. Alemi, Roman Novak, Peter J. Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, and Jeffrey Pennington explore the robust and effective scaling of models from small to large width by adjusting algorithmic and architectural details such as parameterization and optimizer choices. They propose a new perspective on parameterization by challenging key assumptions in prior work regarding the alignment between parameters and data. Through empirical investigations involving tens of thousands of models trained with various optimizers, parameterizations, alignment assumptions, learning rates, and model sizes up to 26.8B parameters,the authors derive new theoretical results under weaker assumptions. Their findings reveal that the best learning rate scaling prescription may have been overlooked in previous studies due to restrictive assumptions. Contrary to existing beliefs that only maximal update parameterization (muP) can achieve hyperparameter transfer,the authors demonstrate that all parameterizations can achieve this feat. Additionally, they introduce a novel per-layer learning rate prescription for standard parameterization that outperforms muP.Furthermore,the researchers highlight the importance of correctly scaling the epsilon parameter in Adam optimizer to prevent gradient underflow.They propose Adam-atan2 as a new numerically stable version of Adam that eliminates the epsilon hyperparameter entirely. The study's comprehensive analysis sheds light on the intricate relationship between parameterization choices and optimizer performance across various model sizes.By challenging conventional wisdom and introducing innovative approaches to hyperparameter tuningand optimization techniques,this research contributes valuable insights to the field of machine learning scalability.
Created on 01 Aug. 2024

Assess the quality of the AI-generated content by voting

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Similar papers summarized with our AI tools

Navigate through even more similar papers through a

tree representation

Look for similar papers (in beta version)

By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.