# A Survey of Large Language Models

**Authors:** Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu  
**arXiv:** [2303.18223](https://arxiv.org/abs/2303.18223)  
**Date:** 2023-03-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.18223v1/

## Key points

- Language is a complex system governed by grammatical rules
- Modeling language has evolved from statistical models to neural models over the past two decades
- Pre-trained language models (PLMs) have been proposed using Transformer models pre-trained over large corpora
- Model scaling can lead to performance improvement when parameters exceed a certain level
- Large language models (LLMs) are PLMs of significant size that show special abilities not present in small-scale language models
- Wayne Xin Zhao et al. review recent advances in LLMs focusing on pre-training, adaptation tuning, utilization, and capacity evaluation
- Pre-training is crucial for achieving high performance on downstream tasks while adaptation tuning aims at fine-tuning PLMs on specific tasks or domains
- Utilization involves using PLMs as building blocks to construct more complex systems such as chatbots or question-answering systems
- Capacity evaluation assesses whether larger models are necessary for specific tasks
- Ethical implications of LLMs include exacerbating existing biases or generating fake news
- Researchers and developers should be aware of these issues and work towards mitigating them.

## Summary

Language is a complex and intricate system of human expressions governed by grammatical rules. Modeling language has been widely studied as an approach for understanding and generating language, evolving from statistical models to neural models over the past two decades. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large corpora, showing strong capabilities in solving various NLP tasks. Researchers have found that model scaling can lead to performance improvement when the parameter scale exceeds a certain level. These enlarged language models not only achieve significant performance improvement but also show some special abilities that are not present in small-scale language models. To distinguish the difference in parameter scale, the research community has coined the term large language models (LLMs) for PLMs of significant size. Wayne Xin Zhao et al. review recent advances in LLMs by introducing background information, key findings, and mainstream techniques focusing on four major aspects: pre-training, adaptation tuning, utilization and capacity evaluation. They summarize available resources for developing LLMs and discuss remaining issues for future directions while highlighting that pre-training is crucial for achieving high performance on downstream tasks while adaptation tuning aims at fine-tuning PLMs on specific tasks or domains. Utilization involves using PLMs as building blocks to construct more complex systems such as chatbots or question-answering systems while capacity evaluation assesses whether larger models are necessary for specific tasks. The authors also discuss ethical implications of LLMs such as their potential to exacerbate existing biases or generate fake news suggesting researchers and developers should be aware of these issues and work towards mitigating them. This survey provides a comprehensive overview of recent advances in LLMs and highlights their potential impact on the AI community serving as a valuable resource for those interested in this rapidly evolving field.