# Optimisation & Generalisation in Networks of Neurons

**Authors:** Jeremy Bernstein  
**arXiv:** [2210.10101](https://arxiv.org/abs/2210.10101)  
**Date:** 2022-10-18  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2210.10101v1/

## Key points

- Jeremy Bernstein's PhD thesis aims to establish theoretical foundations for learning in artificial neural networks.
- The thesis focuses on two key areas: optimisation and generalisation.
- In terms of optimisation, Bernstein proposes a novel theoretical framework that enables the derivation of architecture-dependent first-order optimisation algorithms.
- This approach involves combining a "functional majorisation" of the loss function with "architectural perturbation bounds" that explicitly account for the neural architecture's influence.
- The resulting framework yields optimisation methods that can transfer hyperparameters across different learning problems.
- On the other hand, Bernstein's work on generalisation centres around proposing a new correspondence between ensembles of networks and individual networks.
- This correspondence provides a pathway for transferring PAC-Bayesian generalisation theorems over to individual networks.
- Furthermore, it presents an innovative perspective on how regularisation operates in networks with significantly more parameters than data.
- Overall, this thesis contributes significantly to advancing our understanding of learning in artificial neural networks by developing robust theoretical foundations for both optimisation and generalisation.

## Summary

In his PhD thesis titled "Optimisation & Generalisation in Networks of Neurons," Jeremy Bernstein aims to establish the theoretical foundations for learning in artificial neural networks. The thesis focuses on two key areas: optimisation and generalisation. In terms of optimisation, Bernstein proposes a novel theoretical framework that enables the derivation of architecture-dependent first-order optimisation algorithms. This approach involves combining a "functional majorisation" of the loss function with "architectural perturbation bounds" that explicitly account for the neural architecture's influence. The resulting framework yields optimisation methods that can transfer hyperparameters across different learning problems. On the other hand, Bernstein's work on generalisation centres around proposing a new correspondence between ensembles of networks and individual networks. This correspondence provides a pathway for transferring PAC-Bayesian generalisation theorems over to individual networks. Furthermore, it presents an innovative perspective on how regularisation operates in networks with significantly more parameters than data. Overall, this thesis contributes significantly to advancing our understanding of learning in artificial neural networks by developing robust theoretical foundations for both optimisation and generalisation. By doing so, it opens up new avenues for further research into improving these critical aspects of deep learning models and provides insight into how they can be effectively applied to various tasks.