# Datasets for Large Language Models: A Comprehensive Survey

**Authors:** Yang Liu, Jiahuan Cao, Chongyu Liu, Kai Ding, Lianwen Jin  
**arXiv:** [2402.18041](https://arxiv.org/abs/2402.18041)  
**Date:** 2024-02-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.18041v1/

## Key points

- Large Language Model (LLM) datasets are crucial for the growth and development of LLMs, serving as a foundational infrastructure similar to a root system.
- The paper aims to consolidate and categorize key aspects of LLM datasets from five perspectives: Pre-training Corpora, Instruction Fine-tuning Datasets, Preference Datasets, Evaluation Datasets, and Traditional Natural Language Processing (NLP) Datasets.
- The survey provides a comprehensive review of existing dataset resources across 8 language categories and 32 domains, encompassing statistics from 444 datasets.
- The total data size exceeds 774.5 TB for pre-training corpora and includes 700M instances for other datasets.
- This detailed exploration offers a holistic view of LLM text datasets, serving as a valuable reference point for researchers in their studies.
- The paper contributes to advancing understanding in the domain of large language models and provides related resources available at https://github.com/lmmlzn/Awesome-LLMs-Datasets.

## Summary

In their paper titled "Datasets for Large Language Models: A Comprehensive Survey," authors Yang Liu, Jiahuan Cao, Chongyu Liu, Kai Ding, and Lianwen Jin delve into the pivotal role that Large Language Model (LLM) datasets play in driving the remarkable advancements of LLMs. These datasets are crucial for the growth and development of LLMs, serving as a foundational infrastructure similar to a root system. Recognizing their significance in research, the authors set out to address the lack of a comprehensive overview and thorough analysis of LLM datasets. Their survey aims to consolidate and categorize key aspects of these datasets from five perspectives: Pre-training Corpora, Instruction Fine-tuning Datasets, Preference Datasets, Evaluation Datasets, and Traditional Natural Language Processing (NLP) Datasets. By highlighting current challenges and potential avenues for future investigation, this survey serves as an essential resource for researchers in this field. Additionally,<nl>the authors provide a comprehensive review of existing dataset resources,<nl>encompassing statistics from 444 datasets across 8 language categories and 32 domains. The dataset statistics incorporate information from 20 dimensions,<nl>revealing a total data size exceeding 774.5 TB for pre-training corpora and 700M instances for other datasets.<nl>This detailed exploration presents a holistic view of LLM text datasets,<nl>serving as a valuable reference point for researchers in their studies.<nl>The paper not only contributes to advancing understanding in this domain but also provides related resources available at https://github.com/lmmlzn/Awesome-LLMs-Datasets.<nl>This survey stands as a significant contribution to furthering research in large language models with its thorough analysis and insights into current trends and challenges surrounding LLM datasets.