# Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining

**Authors:** Tianyi Bai, Ling Yang, Zhen Hao Wong, Jiahui Peng, Xinlin Zhuang, Chi Zhang, Lijun Wu, Qiu Jiantao, Wentao Zhang, Binhang Yuan, Conghui He  
**arXiv:** [2410.08102](https://arxiv.org/abs/2410.08102)  
**Date:** 2024-10-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.08102v1/

## Key points

- Efficient data selection is crucial for accelerating the training process of large language models (LLMs).
- Bai et al. introduce a multi-agent collaborative data selection mechanism to address conflicts between different data efficiency methods.
- In their framework, each data selection method acts as an independent agent that dynamically integrates information throughout the LLM training process.
- Extensive empirical studies show significant enhancements in data efficiency, accelerated convergence, and a 10.5% performance gain across multiple language model benchmarks compared to existing methods.
- The collaborative approach streamlines data selection and leverages diverse methodologies to optimize pretraining outcomes for LLMs.

## Summary

In the realm of large language model (LLM) pretraining, efficient data selection plays a pivotal role in accelerating the training process. While numerous methods have been proposed to improve data efficiency, there has been a lack of research addressing the inherent conflicts between these approaches to achieve optimal data selection for LLM pretraining. To address this gap, Bai et al. introduce a groundbreaking multi-agent collaborative data selection mechanism. In their innovative framework, each data selection method functions as an independent agent, with an agent console designed to dynamically integrate information from all agents throughout the LLM training process. Through extensive empirical studies and evaluations, the researchers demonstrate the effectiveness of their multi-agent approach. The experimental results showcase significant enhancements in data efficiency, accelerated convergence during LLM training, and an average performance gain of 10.5% across multiple language model benchmarks compared to existing state-of-the-art methods. The collaborative nature of this novel approach not only streamlines the data selection process but also leverages diverse methodologies to optimize pretraining outcomes for large language models. With contributions from authors such as Tianyi Bai, Ling Yang, Zhen Hao Wong, Jiahui Peng, Xinlin Zhuang, Chi Zhang, Lijun Wu, Qiu Jiantao, Wentao Zhang, Binhang Yuan,and Conghui He; this research represents a significant advancement in enhancing the efficiency and effectiveness of LLM pretraining through collaborative data selection mechanisms.