# Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context

**Authors:** Spencer Frei, Gal Vardi  
**arXiv:** [2410.01774](https://arxiv.org/abs/2410.01774)  
**Date:** 2024-10-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.01774v1/

## Key points

- Researchers investigate transformers as supervised learning algorithms
- Linear transformers show prediction algorithm similar to ordinary least squares for linear regression tasks
- Study focuses on linear transformers trained on random linear classification tasks and gradient descent regularization
- Determining necessary number of pre-training tasks and in-context examples for effective generalization at test-time
- Observing phenomenon of transformer generalizing optimally despite noisy labels in in-context examples
- Study sheds light on behavior, capabilities, generalization abilities, and resilience of trained transformers in classification tasks

## Summary

In this study, the researchers investigate the potential of transformers as supervised learning algorithms. By inputting a set of labeled training examples and an unlabeled test example into the transformer, predictions can be generated for the test example. Previous research has shown that linear transformers exhibit a prediction algorithm similar to ordinary least squares when pre-trained on random instances for linear regression tasks. However, this study focuses on linear transformers trained on random linear classification tasks and delves into the implicit regularization of gradient descent. The aim is to determine the necessary number of pre-training tasks and in-context examples for the transformer to effectively generalize at test-time. This phenomenon is observed when in-context examples are affected by label flipping noise; despite memorizing all examples (including those with noisy labels), the transformer still generalizes optimally for clean test examples. The study sheds light on the behavior and capabilities of trained transformers in classification tasks, providing insights into their generalization abilities and resilience to noisy data.