AI summary · arXiv: 1909.03550v1 · cs.LG
Lecture Notes: Optimization for Machine Learning
The essentials
- Lecture notes authored by Elad Hazan provide a comprehensive overview of optimization techniques tailored for machine learning applications
- Derived from courses at Princeton University, tutorials at MLSS Buenos Aires, and the Simons Foundation in Berkeley
- Thorough analysis of optimization algorithms and their relevance to machine learning models
- Based on cutting-edge research and practical insights from teaching experiences at prestigious institutions
- Topics covered include gradient descent, stochastic optimization, convex optimization, and more
- Real-world examples and case studies included to enhance learning experience
- Valuable resource for students, researchers, and practitioners looking to expand knowledge in optimizing machine learning processes
Key points written by AI: they may contain errors, check them against the paper.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Among the 27 papers around this one, the name that comes back most: Jasper Snoek (2 papers)
3 more people sign several of them. The full map adds their institutions and approaches, from the papers of the last years.
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
Lecture Notes: Convex OptimizationAndreas Habring · 2026 · cited 2 times · Not summarized yet
Abstract
Lecture notes for a course on convex optimization taught by Andreas Habring at Graz University of Technology in 2026. The notes cover mathematical preliminaries, fundamental results about existence and of solutions for minimization problems, projected subgradient descent, proximal-gradient methods, heavy ball gradient descent, Nesterov accelerated gradient descent and FISTA, primal-dual methods, and a short intro to optimal transport.
-
Optimization for machine learningSuvrit Sra, S. Nowozin, Stephen J. Wright · 2010 · cited 738 times · Not summarized yet
-
Advanced Lectures on Machine LearningO. Bousquet, U. von Luxburg, G. Rätsch · 2004 · Lecture Notes in Computer Science · cited 304 times · Not summarized yet
-
Introduction to Machine Learning: Class Notes 67577Amnon Shashua · 2009 · Machine Learning · preprint, not peer reviewed · cited 52 times · AI summary available
Introduction to Machine Learning covers broad statistical, algebraic, spectral, and PAC-learning topics, while these notes focus on optimization for machine learning.
Abstract
Introduction to Machine learning covering Statistical Inference (Bayes, EM, ML/MaxEnt duality), algebraic and spectral methods (PCA, LDA, CCA, Clustering), and PAC learning (the Formal model, VC dimension, Double Sampling theorem).
-
Lecture Notes: Neural Network ArchitecturesEvelyn Herberg · 2023 · Machine Learning · preprint, not peer reviewed · cited 4 times · Not summarized yet
Lecture Notes: Neural Network Architectures treats neural-network training as an optimization problem, applying optimization to specific model architectures.
Abstract
These lecture notes provide an overview of Neural Network architectures from a mathematical point of view. Especially, Machine Learning with Neural Networks is seen as an optimization problem. Covered are an introduction to Neural Networks and the following architectures: Feedforward Neural Network, Convolutional Neural Network, ResNet, and Recurrent Neural Network.
-
Theory of Convex Optimization for Machine LearningSébastien Bubeck · 2014 · arXiv.org · preprint, not peer reviewed · cited 119 times · Not summarized yet
-
Elementos da teoria de aprendizagem de máquina supervisionadaVladimir G. Pestov · 2019 · Machine Learning · preprint, not peer reviewed · cited 2 times · Not summarized yet
Elementos da teoria de aprendizagem de máquina supervisionada teaches foundations of supervised learning, rather than focusing on optimization methods.
Abstract
This is a set of lecture notes for an introductory course (advanced undergaduates or the 1st graduate course) on foundations of supervised machine learning (in Portuguese). The topics include: the geometry of the Hamming cube, concentration of measure, shattering and VC dimension, Glivenko-Cantelli classes, PAC learnability, universal consistency and the k-NN classifier in metric spaces, dimensionality reduction, universal approximation, sample compression. There are appendices on metric and normed spaces, measure theory, etc., making the notes self-contained. Este é um conjunto de notas de aula para um curso introdutório (curso de graduação avançado ou o 1o curso de pós) sobre fundamentos d…
-
Practical Bayesian Optimization of Machine Learning AlgorithmsJasper Snoek, H. Larochelle, Ryan P. Adams · 2012 · Neural Information Processing Systems · cited 10,374 times · Not summarized yet
Abstract
The use of machine learning algorithms frequently involves careful tuning of learning parameters and model hyperparameters. Unfortunately, this tuning is often a "black art" requiring expert experience, rules of thumb, or sometimes brute-force search. There is therefore great appeal for automatic approaches that can optimize the performance of any given learning algorithm to the problem at hand. In this work, we consider this problem through the framework of Bayesian optimization, in which a learning algorithm's generalization performance is modeled as a sample from a Gaussian process (GP). We show that certain choices for the nature of the GP, such as the type of kernel and the treatment of…
Prior work it builds on
-
Input Warping for Bayesian Optimization of Non-Stationary FunctionsJasper Snoek, Kevin Swersky, R. Zemel et al. · 2014 · International Conference on Machine Learning · cited 346 times · Not summarized yet
Abstract
Bayesian optimization has proven to be a highly effective methodology for the global optimization of unknown, expensive and multimodal functions. The ability to accurately model distributions over functions is critical to the effectiveness of Bayesian optimization. Although Gaussian processes provide a flexible prior over functions, there are various classes of functions that remain difficult to model. One of the most frequently occurring of these is the class of non-stationary functions. The optimization of the hyperparameters of machine learning algorithms is a problem domain in which parameters are often manually transformed a priori, for example by optimizing in "log-space," to mitigate …
-
Adafactor: Adaptive Learning Rates with Sublinear Memory CostNoam Shazeer, Mitchell Stern · 2018 · International Conference on Machine Learning · cited 1,412 times · Not summarized yet
Abstract
In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For the case of neural network weight matrices, we propose maintaining only the per-row and per-column sums of these moving averages, and estimating the per-parameter second moments based on these sums. We demonstrate empirically that this method produces similar results to the baseline. Secondly, we show that adaptive methods can produce larger-than-desired updates when the decay …
-
AdaGrad stepsizes: Sharp convergence over nonconvex landscapes, from any initializationRachel A. Ward, Xiaoxia Wu, L. Bottou · 2018 · International Conference on Machine Learning · cited 472 times · Not summarized yet
Abstract
Adaptive gradient methods such as AdaGrad and its variants update the stepsize in stochastic gradient descent on the fly according to the gradients received along the way; such methods have gained widespread use in large-scale optimization for their ability to converge robustly, without the need to fine-tune the stepsize schedule. Yet, the theoretical guarantees to date for AdaGrad are for online and convex optimization. We bridge this gap by providing theoretical guarantees for the convergence of AdaGrad for smooth, nonconvex functions. We show that the norm version of AdaGrad (AdaGrad-Norm) converges to a stationary point at the $\mathcal{O}(\log(N)/\sqrt{N})$ rate in the stochastic settin…
-
Computing Machinery and IntelligenceA. Turing · 1950 · The Philosophy of Artificial Intelligence · cited 12,401 times · Not summarized yet
Papers that cite it
-
Adaptive Gradient Methods with Local GuaranteesZhou Lu, Wen-Han Xia, Sanjeev Arora et al. · 2022 · arXiv.org · preprint, not peer reviewed · cited 13 times · Not summarized yet
Abstract
Adaptive gradient methods are the method of choice for optimization in machine learning and used to train the largest deep models. In this paper we study the problem of learning a local preconditioner, that can change as the data is changing along the optimization trajectory. We propose an adaptive gradient method that has provable adaptive regret guarantees vs. the best local preconditioner. To derive this guarantee, we prove a new adaptive regret bound in online learning that improves upon previous adaptive online learning methods. We demonstrate the robustness of our method in automatically choosing the optimal learning rate schedule for popular benchmarking tasks in vision and language d…
-
Symmetry Teleportation for Accelerated OptimizationBohong Zhao, Nima Dehmamy, R. Walters et al. · 2022 · Neural Information Processing Systems · cited 34 times · Not summarized yet
Abstract
Existing gradient-based optimization methods update parameters locally, in a direction that minimizes the loss function. We study a different approach, symmetry teleportation, that allows parameters to travel a large distance on the loss level set, in order to improve the convergence speed in subsequent steps. Teleportation exploits symmetries in the loss landscape of optimization problems. We derive loss-invariant group actions for test functions in optimization and multi-layer neural networks, and prove a necessary condition for teleportation to improve convergence rate. We also show that our algorithm is closely related to second order methods. Experimentally, we show that teleportation i…
-
Accelerated gradient sliding and variance reduction.D. Dvinskikh, S. Omelchenko, A. Tyurin et al. · 2019 · arXiv: Optimization and Control · preprint, not peer reviewed · cited 1 time · Not summarized yet
Abstract
We consider sum-type strongly convex optimization problem (first term) with smooth convex not proximal friendly composite (second term). We show that the complexity of this problem can be split into optimal number of incremental oracle calls for the first (sum-type) term and optimal number of oracle calls for the second (composite) term. Here under `optimal number' we mean estimate that corresponds to the well known lower bound in the absence of another term.
-
StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language ModelsDing-Zhi Yu, Rui Pan, Yuxing Liu et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 5 times · Not summarized yet
Abstract
Sign-based optimization algorithms, such as SignSGD, have garnered attention for their performance in distributed learning and training large foundation models. Despite their empirical superiority, SignSGD is known to diverge on non-smooth objectives, which are ubiquitous due to ReLUs, max-pools, and mixture-of-experts. To overcome this limitation, we propose StoSignSGD, an algorithm that injects structural stochasticity into the sign operator while maintaining an unbiased update step. In the regime of (online) convex optimization, StoSignSGD rigorously resolves the non-convergence issues of SignSGD, achieving a sharp convergence rate matching the lower bound. For the more challenging non-co…
-
Sketchy: Memory-efficient Adaptive Regularization with Frequent DirectionsVladimir Feinberg, Xin-Yi Chen, Y. Sun et al. · 2023 · Neural Information Processing Systems · cited 23 times · Not summarized yet
Abstract
Adaptive regularization methods that exploit more than the diagonal entries exhibit state of the art performance for many tasks, but can be prohibitive in terms of memory and running time. We find the spectra of the Kronecker-factored gradient covariance matrix in deep learning (DL) training tasks are concentrated on a small leading eigenspace that changes throughout training, motivating a low-rank sketching approach. We describe a generic method for reducing memory and compute requirements of maintaining a matrix preconditioner using the Frequent Directions (FD) sketch. While previous approaches have explored applying FD for second-order optimization, we present a novel analysis which allow…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 6 November 2026.
The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.