# Scaling MLPs: A Tale of Inductive Bias

**Authors:** Gregor Bachmann, Sotiris Anagnostidis, Thomas Hofmann  
**arXiv:** [2306.13575](https://arxiv.org/abs/2306.13575)  
**Date:** 2023-06-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.13575v1/

## Key points

- Authors revisit the multi-layer perceptron (MLP) in deep learning
- Aim to study limits of MLP performance on vision tasks and provide empirical insights
- Understanding MLPs important due to recent narrative suggesting "less inductive bias is better"
- MLPs offer ideal test bed as they are free from any inductive bias
- Experimental data points for MLPs scarce in literature, raising concerns about their role as a proxy
- Authors conduct extensive experiments with MLPs, demonstrating improved performance with scale increase
- Performance of MLPs improves drastically on CIFAR10, CIFAR100, and TinyImageNet vision tasks
- MLPs faithfully mimic behavior of modern counterparts but exhibit stronger or unexpected behaviors in certain components of learning setting
- Computational efficiency makes large pre-training experiments more accessible for academic researchers
- Paper provides valuable insights into performance limits of MLPs on vision tasks and addresses discrepancies between theory and practice

## Summary

In the paper "Scaling MLPs: A Tale of Inductive Bias," authors Gregor Bachmann, Sotiris Anagnostidis, and Thomas Hofmann revisit the multi-layer perceptron (MLP), which is a fundamental building block in deep learning. The authors aim to study the limits of MLP performance on vision tasks and provide empirical insights into its capabilities. The importance of understanding MLPs lies in two key aspects. Firstly, there has been a recent narrative suggesting that "less inductive bias is better," particularly with the rise of transformers overshadowing convolutional models. Therefore, it becomes crucial to explore the boundaries of this hypothesis. MLPs offer an ideal test bed for this exploration as they are completely free from any inductive bias. Secondly, MLPs have predominantly been featured in deep learning theory literature due to their mathematical simplicity, serving as a proxy for explaining empirical phenomena observed in more complex architectures. However, experimental data points for MLPs are scarce in the literature, especially when combined with large pre-training protocols. This discrepancy between theory and practice raises concerns about whether MLPs truly reflect the empirical advances exhibited by practical models or if theorists need to reconsider their role as a proxy. To address these concerns and shed light on both aspects, the authors conduct extensive experiments with MLPs. They demonstrate that as scale increases, the performance of MLPs drastically improves on vision tasks such as CIFAR10 (achieving 93% accuracy), CIFAR100 (achieving 79% accuracy), and TinyImageNet (achieving 69% accuracy). These results highlight that although MLPs lack inductive bias, their performance can be compensated through scaling up their size and complexity. Furthermore, the authors observe that while MLPs faithfully mimic the behavior of modern counterparts they also exhibit stronger or unexpected behaviors in certain components of the learning setting; emphasizing the need to carefully consider these nuances when using them as proxies for understanding more complex architectures. Additionally, they highlight how computational efficiency makes large pre-training experiments more accessible for academic researchers; all experiments were conducted on a single GPU. In summary, this paper provides valuable insights into the performance limits of MLPs on vision tasks and addresses discrepancies between theory and practice; demonstrating that lack of inductive bias can be compensated through scale and emphasizing importance when using them as proxies in deep learning research.