# Small Language Models: Survey, Measurements, and Insights

**Authors:** Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D. Lane, Mengwei Xu  
**arXiv:** [2409.15790](https://arxiv.org/abs/2409.15790)  
**Date:** 2024-09-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2409.15790v1/

## Key points

- Surge in development of small language models (SLMs)
- SLMs based on decoder-only transformer architecture like GPT-2
- Meticulously selected 59 SLMs with open weights and parameter sizes ranging from 100M to 5B
- Focus on base knowledge acquired during pre-training process
- Variability in performance levels across tasks due to differences in hyperparameters and training datasets
- Subjectivity of "small" definition, with a limit of 5B parameters set for SLMs in the study
- Benchmarking inference latency and memory footprints to provide insights for advancing research
- Identification of key innovations and potential research topics for future developments in machine intelligence accessibility, affordability, and efficiency

## Summary

In recent years, there has been a surge in the development of small language models (SLMs), which have garnered increasing attention from both research and industrial communities. These SLMs are based on decoder-only transformer architecture like GPT-2 and are designed for superior performance and real-world deployment. To understand their capabilities and costs, we have meticulously selected 59 SLMs with open weights and parameter sizes ranging from 100M to 5B. Our comprehensive review focuses on the base knowledge acquired during the pre-training process of these SLMs. We exclude fine-tuned models and variants of transformers to maintain consistency in our analysis. While all selected SLMs share similar architectures, they vary in hyperparameters and training datasets, some of which remain closed-source. These differences result in varying performance levels across tasks such as commonsense reasoning, mathematics, coding, and more. It is important to note that the definition of "small" is subjective and relative, with device memory capacities evolving over time. Despite this, we have set a limit of 5B parameters for SLMs in this study. By benchmarking inference latency and memory footprints of these models, we aim to provide valuable insights for advancing research in the field of small language models. Through our exhaustive investigation into these state-of-the-art SLMs, we have identified key innovations and potential research topics that could shape future developments in machine intelligence accessibility, affordability, and efficiency for everyday tasks. By making our results and benchmark tools publicly available, we hope to facilitate further advancements in SLM research and contribute to the broader goal of enhancing machine intelligence capabilities for practical applications.