# Hopfield Networks is All You Need

**Authors:** Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, David Kreil, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter  
**arXiv:** [2008.02217](https://arxiv.org/abs/2008.02217)  
**Date:** 2020-07-16  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2008.02217v1/

## Key points

- The transformer attention mechanism can be seen as the update rule of a new Hopfield network
- The new Hopfield network can store exponentially many patterns and converges with just one update
- There is a trade-off between the number of stored patterns and convergence speed/retrieval error
- The new Hopfield network has three types of energy minima or fixed points: global fixed point, metastable states, and fixed points that store a single pattern
- Transformer and BERT models primarily operate in the global averaging regime in their first layers but switch to metastable states in higher layers
- Learning in transformer and BERT models starts with attention heads that average but most eventually switch to metastable states
- Heads in the last layers steadily learn and appear to use metastable states to collect information from lower layers
- Heads in the last layers are highlighted as promising targets for improving transformers
- Neural networks equipped with Hopfield networks outperform other methods on immune repertoire classification tasks with large numbers of patterns
- A PyTorch layer called "Hopfield" is provided for practical implementation of modern Hopfield networks in deep learning architectures.

## Summary

Keywords: , , , , 

The paper "Hopfield Networks is All You Need" explores the relationship between the transformer attention mechanism and a modern Hopfield network with continuous states. The authors demonstrate that the transformer attention mechanism can be seen as the update rule of this new Hopfield network, which has several advantageous properties. They show that it can store exponentially many patterns relative to its dimension and converges with just one update, exhibiting exponentially small retrieval errors. However, there is a trade-off between the number of stored patterns and convergence speed/retrieval error. The new Hopfield network has three types of energy minima or fixed points: (1) global fixed point averaging over all patterns, (2) metastable states averaging over a subset of patterns, and (3) fixed points that store a single pattern. The authors observe that transformer and BERT models primarily operate in the global averaging regime in their first layers but switch to metastable states in higher layers. They further analyze learning in transformer and BERT models using the Hopfield network interpretation. The authors find that learning starts with attention heads that average but most of them eventually switch to metastable states. However, they note that a majority of heads in the first layers still perform averaging and can be replaced by techniques like their proposed Gaussian weighting. In contrast, heads in the last layers steadily learn and appear to use metastable states to collect information from lower layers. The authors highlight these heads in the last layers as promising targets for improving transformers. They suggest that neural networks equipped with Hopfield networks outperform other methods on immune repertoire classification tasks where several hundreds of thousands of patterns need to be stored. To facilitate practical implementation, the authors provide a new PyTorch layer called "Hopfield" that allows deep learning architectures to incorporate modern Hopfield networks. This integration offers pooling, memory, and attention capabilities within a unified framework. Overall, the authors' work establishes a connection between the transformer attention mechanism and Hopfield networks, shedding light on the learning dynamics and potential improvements for transformers. Their findings provide insights into memory mechanisms in neural networks and offer a powerful concept for enhancing deep learning architectures.