# Training Compute-Optimal Large Language Models

**Authors:** Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae  
**arXiv:** [2203.15556](https://arxiv.org/abs/2203.15556)  
**Date:** 2022-03-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2203.15556v1/

## Key points

- Current large language models are significantly undertrained due to the recent focus on scaling language models while keeping the amount of training data constant.
- For compute-optimal training, both the model size and the number of training tokens should be scaled equally: for every doubling of model size, the number of training tokens should also be doubled.
- Chinchilla, a predicted compute-optimal model with 70 billion parameters and four times more data than Gopher, outperformed Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a wide range of downstream evaluation tasks uniformly and significantly.
- Chinchilla achieved an average accuracy of 67.5% on the MMLU benchmark—a 7% improvement over Gopher's performance.
- Future research should focus not only on scaling up language models but also increasing the amount of training data proportionally to achieve optimal performance within a given computational budget.

## Summary

In their paper titled "Training Compute-Optimal Large Language Models," Jordan Hoffmann and his team investigate the optimal model size and number of tokens required for training a transformer language model under a given compute budget. They discovered that current large language models are significantly undertrained, which is a result of the recent focus on scaling language models while keeping the amount of training data constant. To test their hypothesis, they trained over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens. They found that for compute-optimal training, both the model size and the number of training tokens should be scaled equally: for every doubling of model size, the number of training tokens should also be doubled. The team tested this hypothesis by creating a predicted compute-optimal model named Chinchilla, which uses the same compute budget as Gopher but with 70 billion parameters and four times more data. Chinchilla outperformed Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a wide range of downstream evaluation tasks uniformly and significantly. This means that Chinchilla uses substantially less compute for fine-tuning and inference, making it easier to use downstream. As a highlight, Chinchilla achieved an average accuracy of 67.5% on the MMLU benchmark—a 7% improvement over Gopher's performance. The authors' findings suggest that future research should focus not only on scaling up language models but also increasing the amount of training data proportionally to achieve optimal performance within a given computational budget. Overall, this study provides valuable insights into how to train large language models efficiently while maintaining high performance levels in downstream tasks.