# To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis

**Authors:** Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, Yang You  
**arXiv:** [2305.13230](https://arxiv.org/abs/2305.13230)  
**Date:** 2023-05-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.13230v1/

## Key points

- Dataset size is important in scaling language models
- Large language models (LLMs) are token-hungry during pre-training
- High-quality text data on the web is reaching its scaling limit for LLMs
- Repeating pre-training data for additional epochs can enhance LLMs
- Model is susceptible to overfitting and multi-epoch degradation when repeating pre-training data
- Key factors contributing to multi-epoch degradation include dataset size, model parameters, and training objectives
- Regularization techniques were explored to alleviate multi-epoch degradation, with dropout showing remarkable effectiveness but requiring careful tuning when scaling up the model size
- Leveraging mixture-of-experts (MoE) was discovered as a cost-effective and efficient hyperparameter tuning method for computationally intensive dense LLMs with comparable trainable parameters potentially impacting efficient LLM development on a broader scale.

## Summary

The importance of dataset size in scaling language models has been highlighted by recent research. However, large language models (LLMs) are known to be token-hungry during pre-training, and high-quality text data on the web is reaching its scaling limit for LLMs. To enhance LLMs further, a simple approach is to repeat the pre-training data for additional epochs. In this study by Fuzhao Xue et al., three key aspects were empirically investigated under this approach. Firstly, the consequences of repeating pre-training data were explored, revealing that the model is susceptible to overfitting, leading to multi-epoch degradation. Secondly, the key factors contributing to multi-epoch degradation were examined and found that significant factors include dataset size, model parameters and training objectives while less influential factors consist of dataset quality and model FLOPs. Finally, widely used regularization techniques were explored to alleviate multi-epoch degradation. Most techniques did not yield significant improvements except for dropout which demonstrated remarkable effectiveness but required careful tuning when scaling up the model size. Additionally, leveraging mixture-of-experts (MoE) was discovered as a cost-effective and efficient hyperparameter tuning method for computationally intensive dense LLMs with comparable trainable parameters potentially impacting efficient LLM development on a broader scale. These findings provide insights into improving LLM performance through pre-training data repetition while also highlighting potential pitfalls and solutions for multi-epoch degradation in large scale language modeling tasks.