# Width and Depth Limits Commute in Residual Networks

**Authors:** Soufiane Hayou, Greg Yang  
**arXiv:** [2302.00453](https://arxiv.org/abs/2302.00453)  
**Date:** 2023-02-01  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2302.00453v2/

## Key points

- Study by Soufiane Hayou and Greg Yang on deep neural networks with skip connections
- Scaling branches by $1/\sqrt{depth}$ maintains consistent covariance structure regardless of limit approach
- Increasing width before depth is practical for networks where depth and width are comparable
- Pre-activations in scenario follow Gaussian distributions, impacting Bayesian deep learning
- Theoretical results validated through extensive simulations, showing alignment between theory and practice
- Proof technique establishes that large-depth and large-width limits commute in residual neural networks during initialization
- Concentration of measure result for a McKean-Vlasov process supports analyses prioritizing increasing width before depth
- Technique does not address network behavior post-training, leaving room for exploration into variations based on learning rate selection
- Insights contribute to scalability of deep neural networks with skip connections, emphasizing robustness of covariance structures across dimensions
- Importance of understanding network architecture's influence on model performance and laying foundation for future research on training strategy optimization

## Summary

In their study "Width and Depth Limits Commute in Residual Networks," Soufiane Hayou and Greg Yang investigate the behavior of deep neural networks with skip connections as the width and depth approach infinity. They demonstrate that by scaling branches by $1/\sqrt{depth}$, the covariance structure remains consistent regardless of how the limit is taken. This finding sheds light on why increasing width before depth is still a practical approach for networks where depth and width are comparable. Additionally, Hayou and Yang show that pre-activations in this scenario follow Gaussian distributions, which has implications for Bayesian deep learning. Through extensive simulations, they validate their theoretical results and showcase a strong alignment between theory and practice. The authors employ a novel proof technique to establish that in residual neural networks (resnets), the large-depth and large-width limits commute during initialization. Their concentration of measure result for a McKean-Vlasov process supports previous analyses of deep and wide neural networks that prioritize increasing width before depth. However, they acknowledge that their technique does not address network behavior post-training, leaving room for exploration into potential variations based on learning rate selection. Overall, "Width and Depth Limits Commute in Residual Networks" contributes valuable insights into the scalability of deep neural networks with skip connections, highlighting the robustness of covariance structures across varying dimensions. The study underscores the importance of understanding how network architecture influences model performance and lays a foundation for future research on optimizing training strategies for complex neural networks.