# Instruction Pre-Training: Language Models are Supervised Multitask Learners

**Authors:** Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, Furu Wei  
**arXiv:** [2406.14491](https://arxiv.org/abs/2406.14491)  
**Date:** 2024-06-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.14491v1/

## Key points

- Authors explore supervised multitask pre-training for language models (LMs)
- Introduction of Instruction Pre-Training framework to augment raw corpora with instruction-response pairs
- Utilization of an efficient instruction synthesizer to generate 200 million instruction-response pairs covering over 40 task categories
- Effectiveness of Instruction Pre-Training in enhancing pre-trained base models from scratch demonstrated through experiments
- Consistent improvement of base models and significant benefits from additional instruction tuning observed
- Instruction Pre-Training enables model Llama3-8B to achieve comparable or superior performance compared to Llama3-70B in continual pre-training scenarios
- Availability of authors' model, code, and data for replication or further research at https://github.com/microsoft/LMOps

## Summary

In their paper titled "Instruction Pre-Training: Language Models are Supervised Multitask Learners," authors Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei explore the potential of supervised multitask pre-training for language models (LMs). They highlight the significance of unsupervised multitask pre-training in recent LM advancements but also emphasize the benefits of supervised multitask learning for generalization post-training. To further investigate this concept, they introduce Instruction Pre-Training - a framework designed to efficiently augment large raw corpora with instruction-response pairs for LM pre-training. The authors utilize an efficient instruction synthesizer built on open-source models to generate 200 million instruction-response pairs covering over 40 task categories. Through a series of experiments, they demonstrate the effectiveness of Instruction Pre-Training in enhancing pre-trained base models from scratch. Notably, they find that this approach consistently improves base models and shows even more significant benefits from additional instruction tuning. Additionally, in continual pre-training scenarios, Instruction Pre-Training enables their model Llama3-8B to achieve comparable or even superior performance compared to Llama3-70B. This highlights the scalability and efficacy of their proposed framework in improving LM performance through supervised multitask learning. For those interested in replicating or building upon their work, the authors have made their model, code, and data available at https://github.com/microsoft/LMOps.