# Are All Layers Created Equal?

**Authors:** Chiyuan Zhang, Samy Bengio, Yoram Singer  
**arXiv:** [1902.01996](https://arxiv.org/abs/1902.01996)  
**Date:** 2019-02-06  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/1902.01996v1/

## Key points

- Understanding how neural networks learn and generalize is a key focus in deep learning research
- Large, over-parameterized networks challenge traditional notions of uniform convergence and learnability
- Study by Chiyuan Zhang, Samy Bengio, and Yoram Singer investigates layer-wise functional structure of deep neural networks
- Layers can be classified as "robust" or "critical"
- Critical layers are sensitive to changes and resetting them can negatively impact model performance
- Robust layers show resilience to resets and remain stable throughout training
- The study highlights the complexity of deep neural networks and challenges simplistic approaches when studying generalization

## Summary

, , , , 

In the realm of deep learning, understanding how neural networks learn and generalize has been a key focus of research in recent years. The success of large, over-parameterized networks has challenged traditional notions of uniform convergence and learnability, prompting researchers to delve deeper into the inner workings of these complex models. In their study titled "Are All Layers Created Equal? ", authors Chiyuan Zhang, Samy Bengio, and Yoram Singer empirically investigate the layer-wise functional structure of deep neural networks. They introduce the concept of post-training re-initialization and re-randomization robustness to explore the characteristics of different layers within these models. Through their research, they uncover evidence that layers can be classified as either "robust" or "critical". Critical layers are sensitive to changes and resetting them to their initial values can have negative consequences on model performance. In contrast, robust layers show resilience to such resets and often remain stable throughout training. This distinction between robust and critical layers highlights the complexity inherent in deep neural networks and challenges simplistic approaches like parameter counting or norm accounting when studying generalization in these models. By shedding light on the heterogeneous nature of layers within deep architectures, this study contributes valuable insights towards unraveling the mysteries of learning and generalization in deep learning systems.