# Teaching Arithmetic to Small Transformers

**Authors:** Nayoung Lee, Kartik Sreenivasan, Jason D. Lee, Kangwook Lee, Dimitris Papailiopoulos  
**arXiv:** [2307.03381](https://arxiv.org/abs/2307.03381)  
**Date:** 2023-07-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.03381v1/

## Key points

- Small transformers can learn arithmetic operations using the next-token prediction objective
- Large language models like GPT-4 can exhibit emergent capabilities in basic arithmetic tasks
- Conventional training data is not effective for learning arithmetic, but simple formatting changes can improve accuracy
- Training small transformers on chain-of-thought style data greatly improves accuracy, sample complexity, and convergence speed
- The interplay between arithmetic and text data during training is investigated, along with few-shot prompting, pretraining, and model scale
- Challenges related to length generalization are discussed
- High quality instructive data that considers the specific characteristics of the next word prediction objective is important for eliciting arithmetic capabilities in small transformers.

## Summary

In their study titled "Teaching Arithmetic to Small Transformers," authors Nayoung Lee, Kartik Sreenivasan, Jason D. Lee, Kangwook Lee, and Dimitris Papailiopoulos explore how small transformers can learn arithmetic operations such as addition, multiplication, and square root using the next-token prediction objective. They find that large language models like GPT-4 can exhibit emergent capabilities in general-purpose tasks like basic arithmetic when trained on extensive text data. However, these tasks are not explicitly encoded by the unsupervised objective of next-token prediction. The authors begin by demonstrating that conventional training data is not the most effective for learning arithmetic. They show that simple formatting changes can significantly improve accuracy and lead to sharp phase transitions based on the scale of training data. In some cases, these transitions can be explained through connections to low-rank matrix completion. Building on previous work, the authors train small transformers from random initialization on chain-of-thought style data that includes intermediate step results. Remarkably, even without pretraining this approach greatly improves accuracy, sample complexity and convergence speed. The study also investigates the interplay between arithmetic and text data during training and examines the effects of few-shot prompting, pretraining and model scale. Additionally it discusses challenges related to length generalization. Overall this research highlights the importance of high quality instructive data that considers the specific characteristics of the next word prediction objective for effectively eliciting arithmetic capabilities in small transformers.