# Transformers learn in-context by gradient descent

**Authors:** Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, Max Vladymyrov  
**arXiv:** [2212.07677](https://arxiv.org/abs/2212.07677)  
**Date:** 2022-12-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.07677v1/

## Key points

- Transformers have emerged as the leading neural network architecture in various domains of machine learning.
- Transformers can transfer knowledge and learn in-context with limited examples.
- Training Transformers on auto-regressive tasks is closely related to gradient-based meta-learning formulations.
- Self-attention-only Transformers exhibit remarkable similarity to models learned by gradient descent when trained on simple regression tasks.
- Trained Transformers implement gradient descent in their forward pass, as evidenced by the weights obtained through optimization matching the weight construction.
- Transformers surpass plain gradient descent by employing an iterative curvature correction mechanism.
- They learn linear models on deep data representations to effectively solve non-linear regression tasks.
- This mechanism bears similarities to induction-head learning, which is crucial for in-context learning.

## Summary

Transformers have emerged as the leading neural network architecture in various domains of machine learning. Their ability to transfer knowledge and learn in-context with limited examples has made them highly sought after. However, the underlying mechanisms that enable Transformers to become in-context learners are not well understood. In this study, the authors propose that training Transformers on auto-regressive tasks can be closely related to established gradient-based meta-learning formulations. They present a simple weight construction that demonstrates the equivalence between data transformations induced by a single linear self-attention layer and gradient descent (GD) on a regression loss. Empirical evidence shows that when trained on simple regression tasks, self-attention-only Transformers exhibit remarkable similarity to models learned by GD. The weights obtained through optimization also match the weight construction, suggesting that trained Transformers implement gradient descent in their forward pass. This provides a mechanistic understanding of how optimized Transformers learn in-context for regression problems. The study further identifies how Transformers surpass plain gradient descent by employing an iterative curvature correction mechanism. It is demonstrated that they learn linear models on deep data representations to effectively solve non-linear regression tasks. Interestingly, this mechanism bears similarities to induction-head learning which is crucial for in-context learning. Overall, this research sheds light on how trained Transformers leverage gradient descent during their forward pass to achieve in-context learning. It offers insights into the inner workings of optimized Transformers and highlights their ability to go beyond traditional gradient descent methods through iterative curvature correction and deep data representation learning. The findings also establish connections between these mechanisms and induction-head learning within Transformers.