Breaking the Curse of Dimensionality in Deep Neural Networks by Learning Invariant Representations

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Close Not relevant

   The Modern Mathematics of Deep Learning

   Julius Berner, Philipp Grohs, Gitta Kutyniok et al. · 2021 · Mathematical Aspects of Deep Learning, pp. 1-111. Cambridge University Press, 2022 · AI summary available

   Abstract

   We describe the new field of mathematical analysis of deep learning. This field emerged around a list of research questions that were not answered within the classical framework of learning theory. These questions concern: the outstanding generalization power of overparametrized neural networks, the role of depth in deep architectures, the apparent absence of the curse of dimensionality, the surprisingly successful optimization performance despite the non-convexity of the problem, understanding what features are learned, why deep architectures perform exceptionally well in physical problems, and which fine aspects of an architecture affect the behavior of a learning task in which way. We pre…

2. Close Not relevant

   Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges

   Michael M. Bronstein, Joan Bruna, Taco Cohen et al. · 2021 · Machine Learning · AI summary available

   Abstract

   The last decade has witnessed an experimental revolution in data science and machine learning, epitomised by deep learning methods. Indeed, many high-dimensional learning tasks previously thought to be beyond reach -- such as computer vision, playing Go, or protein folding -- are in fact feasible with appropriate computational scale. Remarkably, the essence of deep learning is built from two simple algorithmic principles: first, the notion of representation or feature learning, whereby adapted, often hierarchical, features capture the appropriate notion of regularity for each task, and second, learning by local gradient-descent type methods, typically implemented as backpropagation. While le…

3. Close Not relevant

   The Principles of Deep Learning Theory

   Daniel A. Roberts, Sho Yaida, Boris Hanin · 2021 · Machine Learning · Not summarized yet

   Abstract

   This book develops an effective theory approach to understanding deep neural networks of practical relevance. Beginning from a first-principles component-level picture of networks, we explain how to determine an accurate description of the output of trained networks by solving layer-to-layer iteration equations and nonlinear learning dynamics. A main result is that the predictions of networks are described by nearly-Gaussian distributions, with the depth-to-width aspect ratio of the network controlling the deviations from the infinite-width Gaussian description. We explain how these effectively-deep networks learn nontrivial representations from training and more broadly analyze the mechanis…

4. Close Not relevant

   Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth

   Thao Nguyen, Maithra Raghu, Simon Kornblith · 2020 · Machine Learning · AI summary available

   Abstract

   A key factor in the success of deep neural networks is the ability to scale models to improve performance by varying the architecture depth and width. This simple property of neural network design has resulted in highly effective architectures for a variety of tasks. Nevertheless, there is limited understanding of effects of depth and width on the learned representations. In this paper, we study this fundamental question. We begin by investigating how varying depth and width affects model hidden representations, finding a characteristic block structure in the hidden representations of larger capacity (wider or deeper) models. We demonstrate that this block structure arises when model capacit…

5. Close Not relevant

   Deep Learning is Not So Mysterious or Different

   Andrew Gordon Wilson · 2025 · Machine Learning · AI summary available

   Abstract

   Deep neural networks are often seen as different from other model classes by defying conventional notions of generalization. Popular examples of anomalous generalization behaviour include benign overfitting, double descent, and the success of overparametrization. We argue that these phenomena are not distinct to neural networks, or particularly mysterious. Moreover, this generalization behaviour can be intuitively understood, and rigorously characterized using long-standing generalization frameworks such as PAC-Bayes and countable hypothesis bounds. We present soft inductive biases as a key unifying principle in explaining these phenomena: rather than restricting the hypothesis space to avoi…

6. Close Not relevant

   Relational inductive biases, deep learning, and graph networks

   Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst et al. · 2018 · Machine Learning · AI summary available

   Abstract

   Artificial intelligence (AI) has undergone a renaissance recently, making major progress in key domains such as vision, language, control, and decision-making. This has been due, in part, to cheap data and cheap compute resources, which have fit the natural strengths of deep learning. However, many defining characteristics of human intelligence, which developed under much different pressures, remain out of reach for current approaches. In particular, generalizing beyond one's experiences--a hallmark of human intelligence from infancy--remains a formidable challenge for modern AI. The following is part position paper, part review, and part unification. We argue that combinatorial generalizati…

7. Close Not relevant

   Opening the black box of deep learning

   Dian Lei, Xiaoxiao Chen, Jianfei Zhao · 2018 · Machine Learning · AI summary available

   Abstract

   The great success of deep learning shows that its technology contains profound truth, and understanding its internal mechanism not only has important implications for the development of its technology and effective application in various fields, but also provides meaningful insights into the understanding of human brain mechanism. At present, most of the theoretical research on deep learning is based on mathematics. This dissertation proposes that the neural network of deep learning is a physical system, examines deep learning from three different perspectives: microscopic, macroscopic, and physical world views, answers multiple theoretical puzzles in deep learning by using physics principle…

8. Close Not relevant

   Why & When Deep Learning Works: Looking Inside Deep Learnings

   Ronny Ronen · 2017 · Machine Learning · Not summarized yet

   Abstract

   The Intel Collaborative Research Institute for Computational Intelligence (ICRI-CI) has been heavily supporting Machine Learning and Deep Learning research from its foundation in 2012. We have asked six leading ICRI-CI Deep Learning researchers to address the challenge of "Why & When Deep Learning works", with the goal of looking inside Deep Learning, providing insights on how deep networks function, and uncovering key observations on their expressiveness, limitations, and potential. The output of this challenge resulted in five papers that address different facets of deep learning. These different facets include a high-level understating of why and when deep networks work (and do not work),…