# On the infinite-depth limit of finite-width neural networks

**Authors:** Soufiane Hayou  
**arXiv:** [2210.00688](https://arxiv.org/abs/2210.00688)  
**Date:** 2022-10-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2210.00688v3/

## Key points

- Authors investigate infinite-depth limit of finite-width residual neural networks with random Gaussian weights
- Pre-activations converge to zero-drift diffusion process in infinite-depth limit, different from weak convergence to Gaussian random variable in infinite-width limit
- Diverse distributions observed in infinite-depth limit based on activation function chosen, with distinct closed-form expressions identified in two scenarios
- Change in regime for post-activation norms when transitioning from width of 3 to 4
- Comparison between sequential limits: first approaching infinite depth then width versus more common path of width preceding depth
- Understanding limiting laws crucial for designing resilient neural networks and gaining insights into overparameterized networks
- Unlike infinite-width limit where Gaussian distribution is typically obtained under certain conditions, behavior in infinite-depth limit influenced by activation function selection
- Leveraging Itˆo's lemma allows obtaining known distributions by adjusting activation functions carefully
- Zero probability of process collapse for specific conditions in infinite-depth neural networks with various activation functions (including ReLU), highlighting robustness and stability compared to other configurations

## Summary

The authors of this paper investigate the infinite-depth limit of finite-width residual neural networks with random Gaussian weights. By fixing the width and allowing the depth to approach infinity, they demonstrate that the pre-activations converge in distribution to a zero-drift diffusion process through proper scaling. This is different from the infinite-width limit where pre-activations tend to weakly converge to a Gaussian random variable. The infinite-depth limit also yields diverse distributions depending on the chosen activation function. Two scenarios are identified where these distributions have distinct closed-form expressions. Interestingly, there is a change in regime for post-activation norms when transitioning from a width of 3 to 4. The study also compares two sequential limits: first approaching infinite depth followed by infinite width versus the more commonly explored path of infinite width preceding infinite depth. Understanding these limiting laws is crucial for designing resilient neural networks and gaining insights into overparameterized networks. While previous research has focused on the infinite-width limit, this work contributes by investigating the infinite-depth limit of finite-width neural networks. A key finding is that unlike in the case of infinite width where a Gaussian distribution is typically obtained under certain conditions regarding activation functions, behavior in the infinite-depth limit is notably influenced by activation function selection. Leveraging Itˆo's lemma allows for obtaining known distributions by carefully adjusting activation functions. Furthermore, an important characteristic is revealed regarding general width limits: there is a zero probability of process collapse for specific conditions in infinite-depth neural networks with various activation functions (including ReLU). This highlights their robustness and stability compared to other configurations.