# How to Train Long-Context Language Models (Effectively)

**Authors:** Tianyu Gao, Alexander Wettig, Howard Yen, Danqi Chen  
**arXiv:** [2410.02660](https://arxiv.org/abs/2410.02660)  
**Date:** 2024-10-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.02660v1/

## Key points

- Study focuses on continued training and supervised fine-tuning of language models (LM) for long-context information
- Establishes robust evaluation protocol using diverse long-context tasks post-SFT with instruction data
- Experimentation reveals importance of optimal data mix, selection of instruction tuning datasets, and leveraging sources like code repositories and books
- Training with sequence length exceeding evaluation length enhances long-context performance
- Using only short instruction datasets during SFT can lead to strong performance on long-context tasks
- Introduction of ProLong-8B model surpasses previous models in long-context performance at a length of 128K tokens, showcasing exceptional processing capabilities up to 512K tokens

## Summary

In their study titled "How to Train Long-Context Language Models (Effectively)," authors Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen delve into the realm of continued training and supervised fine-tuning (SFT) of language models (LM) to harness the power of long-context information. The researchers establish a robust evaluation protocol that moves beyond traditional metrics like perplexity or simple needle-in-a-haystack tests, opting instead for a diverse set of long-context tasks. By evaluating models post-SFT with instruction data, they are able to better gauge long-context capabilities. Through meticulous experimentation, the team explores various aspects such as the optimal data mix for continued pre-training and the selection of instruction tuning datasets. They discover that leveraging sources like code repositories and books can provide valuable long data but emphasize the importance of complementing them with high-quality short data. Additionally, they find that training with a sequence length exceeding the evaluation length significantly enhances long-context performance. The study reveals that using only short instruction datasets during SFT can lead to strong performance on long-context tasks. The culmination of their efforts is ProLong-8B, an advanced model initialized from Llama-3 and trained on 40B tokens. ProLong showcases state-of-the-art long-context performance among models of similar size at a length of 128K, surpassing Llama-3.18B-Instruct on most long-context tasks despite being exposed to only 5% as many tokens during training. Moreover, ProLong demonstrates exceptional processing capabilities by effectively handling up to 512K tokens, boasting one of the longest context windows among publicly available LM's. The findings from this study offer valuable insights into optimizing training strategies for language models to excel in capturing and utilizing extensive contextual information effectively.