# A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends

**Authors:** Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, Jiachi Chen  
**arXiv:** [2311.10372](https://arxiv.org/abs/2311.10372)  
**Date:** 2023-11-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.10372v2/

## Key points

- The study is titled "A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends"
- The authors conduct a comprehensive survey and analysis of specialized language models for software engineering tasks known as Code LLMs
- What are the specific LLMs designed for software engineering tasks and how do they relate to each other?
- Do Code LLMs outperform general LLMs in software engineering tasks?
- Which LLMs excel in different software engineering tasks?
- They collect relevant literature and work from major databases and open-source communities resulting in 134 works for analysis
- The Code LLMs are categorized based on their publishers and examined for relationships with general LLMs and among themselves
- Performance differences between general LLMs and Code LLMs in various software engineering tasks are investigated to demonstrate the impact of base models and Code LLMs
- The study focuses on maintaining the performance of LLMs across multiple mainstream benchmarks to identify the best-performing models for each software engineering task
- The research assists developers of Code LLMs in choosing suitable base models for developing advanced versions
- Provides valuable insights for practitioners to better understand key improvement directions for Code LLMs
- Contributes to bridging the gap in systematic investigation into Code LLMs and their performance
- Offers valuable guidance for both developers and practitioners in the field of software engineering

## Summary

In their study titled "A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends," authors Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye and Jiachi Chen conduct a comprehensive survey and analysis of specialized language models designed specifically for software engineering tasks known as Code LLMs. The authors aim to address three key questions: (1) What are the specific LLMs designed for software engineering tasks and how do they relate to each other? (2) Do Code LLMs outperform general LLMs in software engineering tasks? (3) Which LLMs excel in different software engineering tasks? To answer these questions, the authors collect relevant literature and work from major databases and open-source communities resulting in 134 works for analysis. They categorize the Code LLMs based on their publishers and examine their relationships with general LLMs as well as among themselves. Furthermore, they investigate the performance differences between general LLMs and Code LLMs in various software engineering tasks to demonstrate the impact of base models and Code LLMs. The study also focuses on maintaining the performance of LLMs across multiple mainstream benchmarks to identify the best-performing models for each software engineering task. This research not only assists developers of Code LLMs in choosing suitable base models for developing advanced versions but also provides valuable insights for practitioners to better understand key improvement directions for Code LLMs. Overall, this study contributes to bridging the gap in systematic investigation into Code LLMs and their performance. By providing a comprehensive overview of different types of Code LLMs and analyzing their performance compared to general LLMs, it offers valuable guidance for both developers and practitioners in the field of software engineering.