# Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

**Authors:** Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, Vedant Misra  
**arXiv:** [2201.02177](https://arxiv.org/abs/2201.02177)  
**Date:** 2022-01-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2201.02177v1/

## Key points

- Authors study generalization in neural networks using small algorithmically generated datasets
- Neural networks undergo a process of "grokking" patterns within data, leading to significant improvements in generalization performance
- Generalization can occur even after overfitting
- Generalization varies with dataset size, smaller datasets require more optimization for effective generalization
- Small algorithmic datasets offer an ideal platform for investigating the ability of overparametrized neural networks to generalize beyond mere memorization
- Research sheds light on mechanisms underlying neural network generalization and emphasizes studying diverse dataset sizes for optimizing model performance

## Summary

In the paper titled "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets," authors Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra delve into the study of generalization in neural networks using small algorithmically generated datasets. This unique setting allows for a detailed examination of data efficiency, memorization, generalization, and the speed of learning. The authors demonstrate that neural networks undergo a process of "grokking" patterns within the data, leading to significant improvements in generalization performance from random chance levels to perfect generalization. Surprisingly, this enhancement in generalization can occur even after the point of overfitting. Furthermore, the study explores how generalization varies with dataset size, revealing that smaller datasets necessitate increasing levels of optimization for effective generalization. The authors argue that these small algorithmic datasets offer an ideal platform for investigating a complex and poorly understood aspect of deep learning: the ability of overparametrized neural networks to generalize beyond mere memorization of the finite training dataset. This research not only sheds light on the mechanisms underlying neural network generalization but also underscores the importance of studying diverse dataset sizes to gain deeper insights into optimizing model performance. The findings presented in this paper have significant implications for advancing our understanding of deep learning processes and improving model generalizability in practical applications.