# Large-Scale Data Selection for Instruction Tuning

**Authors:** Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, Pradeep Dasigi  
**arXiv:** [2503.01807](https://arxiv.org/abs/2503.01807)  
**Date:** 2025-03-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2503.01807v1/

## Key points

- Importance of selecting high-quality training data for instruction-tuning language models
- Carefully curated datasets produce models that outperform those trained on larger, noisier datasets
- Automated data selection approaches and their effectiveness on large-scale datasets
- Evaluation of scalability of methods using up to 2.5 million samples from pools as large as 5.8 million samples across seven diverse tasks
- Representation-based data selection (RDS+) consistently outperformed more complex methods while being more compute-efficient

## Summary

, , , , 

In the study "Large-Scale Data Selection for Instruction Tuning," conducted by Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi, the importance of selecting high-quality training data for instruction-tuning language models is emphasized. The researchers highlight how carefully curated datasets have been shown to produce models that outperform those trained on larger, noisier datasets. This emphasizes the crucial role of data selection in achieving optimal performance in language models. The study focuses on automated data selection approaches and their effectiveness when applied to large-scale datasets. While these methods are typically tested on small datasets from small pools, popular deployed instruction-tuned models often train on hundreds of thousands to millions of samples from even larger data pools. To evaluate the scalability of these methods, the researchers conducted a systematic study using up to 2.5 million samples from pools as large as 5.8 million samples across seven diverse tasks. The results showed that many proposed methods fell short of random selection in this setting and even performed worse when given access to larger pools of data. However, a variant of representation-based data selection (RDS+), which utilizes weighted mean pooling of pretrained LM hidden states, consistently outperformed more complex methods while being more compute-efficient. These findings highlight the importance of examining the scaling properties of automated selection methods and provide valuable insights into the challenges and opportunities in large-scale data selection for instruction tuning in language models. The researchers have made their code, data, and models available for further exploration. Additionally, they extended their experimental design using TÜLU 2 as a base dataset and model family and reported results using TÜLU 3 mixture and Llama 3.1 in subsequent sections. Further details on the size and makeup of data pools considered in the study are provided in Figure 2, with additional information available in Appendices B and E.