# A Survey of Large Language Models

**Authors:** Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu  
**arXiv:** [2303.18223](https://arxiv.org/abs/2303.18223)  
**Date:** 2023-03-31  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2303.18223v10/

## Key points

- Language modeling in artificial intelligence is a key research area for enhancing machines' understanding and generation of human language.
- Pre-trained language models (PLMs) developed by training Transformer models on extensive data sets have shown remarkable proficiency in natural language processing tasks.
- Increasing model size significantly boosts performance levels, leading to the development of large language models (LLMs) with unique capabilities beyond conventional limits.
- The launch of ChatGPT has marked a significant milestone in the field, attracting widespread interest from society and revolutionizing AI algorithm development and utilization.
- This comprehensive survey focuses on pre-training techniques, adaptation tuning strategies, utilization methods, and capacity evaluation mechanisms for large language models (LLMs).
- The exploration and advancement of LLMs have garnered substantial attention from academic and industrial circles, shaping future research directions.

## Summary

In the realm of artificial intelligence, language modeling has emerged as a pivotal area of research, aiming to enhance machines' ability to comprehend and generate human language. Over the past two decades, there has been a notable shift from traditional statistical language models to more advanced neural language models. One of the recent breakthroughs in this field is the development of pre-trained language models (PLMs), which involve training Transformer models on extensive corpora data sets. These PLMs have showcased remarkable proficiency in tackling various natural language processing (NLP) tasks. Researchers have observed that scaling up model size can significantly boost performance levels. This led to further investigations into the impact of increasing model parameters beyond conventional limits. Surprisingly, when these enlarged language models surpass a certain threshold, they not only exhibit enhanced performance but also demonstrate unique capabilities absent in smaller-scale models. To distinguish these high-parameter scale models, they are now commonly referred to as large language models (LLMs). The exploration and advancement of LLMs have garnered substantial attention from both academic and industrial circles. Notably, the launch of ChatGPT has marked a significant milestone in this domain, attracting widespread interest from society at large. The continuous evolution of LLMs is poised to revolutionize how AI algorithms are developed and utilized across various applications. This comprehensive survey delves into recent advancements in LLMs by providing insights into their background, key discoveries, and prevalent methodologies. It focuses on four core aspects: pre-training techniques, adaptation tuning strategies, utilization methods, and capacity evaluation mechanisms. Additionally, it outlines available resources for developing LLMs while addressing lingering challenges that may shape future research directions. Authored by a diverse team comprising Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min,
Beichen Zhang,
Junjie Zhang,
Zican Dong,
Yifan Du,
Chen Yang,
Yushuo Chen,
Zhipeng Chen,
Jinhao Jiang,
Ruiyang Ren,
Yifan Li,
Xinyu Tang,
Zikang Liu
and others; this survey encapsulates the cutting-edge developments in LLMs and underscores their transformative potential within the AI community.