# The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents

**Authors:** Yatin Dandi, Emanuele Troiani, Luca Arnaboldi, Luca Pesce, Lenka Zdeborová, Florent Krzakala  
**arXiv:** [2402.03220](https://arxiv.org/abs/2402.03220)  
**Date:** 2024-02-05  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2402.03220v1/

## Key points

- Dandi et al. investigate the training dynamics of two-layer neural networks when learning multi-index target functions.
- Multi-pass gradient descent (GD) that reuses batches multiple times significantly alters conclusions about learnable functions compared to single-pass GD.
- Multi-pass GD with a finite step size overcomes limitations imposed by gradient flow and single-pass GD.
- Reusing batches allows for achieving overlap with the target subspace in just two time steps, even for functions without the staircase property.
- The authors employ Dynamical Mean-Field Theory (DMFT) analysis and provide a closed-form description of the dynamical process of low-dimensional projections of weights.
- Numerical experiments are conducted to illustrate the theoretical findings.
- This study expands our understanding of training dynamics in neural networks and highlights the benefits of reusing batches in multi-pass GD.
- The results have implications for improving learning algorithms and expanding the class of efficiently learnable functions in finite time.

## Summary

In their study, Dandi et al. investigate the training dynamics of two-layer neural networks when learning multi-index target functions. Specifically focusing on multi-pass gradient descent (GD) that reuses batches multiple times, they demonstrate that this approach significantly alters conclusions about learnable functions compared to single-pass GD. The authors find that multi-pass GD with a finite step size overcomes limitations imposed by gradient flow and single-pass GD, as determined by the information exponent and leap exponent of the target function. This is a significant finding as it suggests that reusing batches allows for achieving overlap with the target subspace in just two time steps, even for functions without the staircase property. To support their claims, Dandi et al. employ Dynamical Mean-Field Theory (DMFT) analysis and provide a closed-form description of the dynamical process of low-dimensional projections of weights. They also conduct numerical experiments to illustrate their theoretical findings. Overall, this study expands our understanding of training dynamics in neural networks and highlights the benefits of reusing batches in multi-pass GD. These results have implications for improving learning algorithms and expanding the class of efficiently learnable functions in finite time.