# Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster

**Authors:** Nolan Dey, Gurpreet Gosal,  Zhiming,  Chen, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness  
**arXiv:** [2304.03208](https://arxiv.org/abs/2304.03208)  
**Date:** 2023-04-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.03208v1/

## Key points

- The paper explores recent research advancements that improve large language models through efficient pre-training and scaling, open datasets, and tools.
- Cerebras-GPT is introduced as a family of open compute-optimal language models scaled from 111M to 13B parameters by combining these advancements.
- Cerebras-GPT models are trained on the Eleuther Pile dataset following DeepMind Chinchilla scaling rules for efficient pre-training.
- Predictable power-law scaling is characterized in this study, and Cerebras-GPT is compared with other publicly available models to demonstrate state-of-the-art training efficiency on both pre-training and downstream objectives.
- Maximal Update Parameterization ($\mu$P) can further improve large model scaling by enhancing accuracy and hyperparameter predictability at scale.
- This paper is the first open and reproducible work comparing compute-optimal model scaling to models trained on fixed dataset sizes.
- The authors release their pre-trained models and code, making them available on HuggingFace for researchers to build upon.

## Summary

In their paper titled "Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster," Nolan Dey and his team explore recent research advancements that improve large language models through efficient pre-training and scaling, open datasets, and tools. They introduce Cerebras-GPT, a family of open compute-optimal language models scaled from 111M to 13B parameters by combining these advancements. The authors train Cerebras-GPT models on the Eleuther Pile dataset following DeepMind Chinchilla scaling rules for efficient pre-training, which ensures the highest accuracy for a given compute budget. The predictable power-law scaling is characterized in this study, and Cerebras-GPT is compared with other publicly available models to demonstrate that all Cerebras-GPT models have state-of-the-art training efficiency on both pre-training and downstream objectives. The authors also describe their learnings about how Maximal Update Parameterization ($\mu$P) can further improve large model scaling by enhancing accuracy and hyperparameter predictability at scale. This paper is the first open and reproducible work comparing compute-optimal model scaling to models trained on fixed dataset sizes. The authors release their pre-trained models and code, making them available on HuggingFace for researchers to build upon. Overall, this study provides valuable insights into improving large language models' efficiency through efficient pre-training and scaling while offering an open source solution.