# Approaching Deep Learning through the Spectral Dynamics of Weights

**Authors:** David Yunis, Kumar Kshitij Patel, Samuel Wheeler, Pedro Savarese, Gal Vardi, Karen Livescu, Michael Maire, Matthew R. Walter  
**arXiv:** [2408.11804](https://arxiv.org/abs/2408.11804)  
**Date:** 2024-08-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2408.11804v1/

## Key points

- Yunis et al. propose an empirical approach focusing on the spectral dynamics of weights in deep learning optimization.
- The authors analyze singular values and vectors to unify and clarify various phenomena observed in deep learning models during optimization.
- A consistent bias in optimization processes is identified, which is enhanced by weight decay beyond its traditional function as a norm regularizer.
- Spectral dynamics of weights can distinguish between memorizing networks and generalizing ones, offering a new perspective on this issue in neural network research.
- The authors investigate the emergence of well-performing sparse subnetworks (lottery tickets) using spectral dynamics and analyze loss surface structures through linear mode connectivity.
- Understanding spectral dynamics provides a coherent framework for interpreting neural network behaviors across diverse settings, bridging gaps between different approaches in deep learning research.

## Summary

In their paper titled "Approaching Deep Learning through the Spectral Dynamics of Weights," Yunis et al. (2022) propose an empirical approach that focuses on the spectral dynamics of weights in deep learning optimization. The authors aim to unify and clarify various phenomena observed in deep learning models by analyzing the behavior of singular values and vectors during optimization. Through a series of experiments ranging from small-scale tasks to large-scale applications such as image classification, image generation, speech recognition, and language modeling, they identify a consistent bias in optimization processes. This bias is enhanced by weight decay beyond its traditional function as a norm regularizer. Furthermore, Yunis et al. demonstrate that the spectral dynamics of weights can distinguish between memorizing networks and generalizing ones, providing a fresh perspective on this long-standing issue in neural network research. Additionally, the authors leverage these spectral dynamics to investigate the emergence of well-performing sparse subnetworks (known as lottery tickets) and analyze the structure of loss surfaces through linear mode connectivity. Their findings suggest that understanding spectral dynamics offers a coherent framework for interpreting neural network behaviors across diverse settings. By bridging gaps between different approaches in deep learning research, this work provides valuable insights into optimizing deep learning models effectively and sheds light on key factors influencing model performance.