# Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation

**Authors:** Kailun Jin, Chung-Yu Wang, Hung Viet Pham, Hadi Hemmati  
**arXiv:** [2402.11702](https://arxiv.org/abs/2402.11702)  
**Date:** 2024-02-18  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2402.11702v2/

## Key points

- Large Language Models (LLMs) are powerful tools for code generation in Artificial Intelligence (AI)
- LLMs demonstrate proficiency in code completion, source code mapping, and system maintenance
- Current evaluations of LLMs have mainly been in research settings, highlighting a gap in understanding their effectiveness in real-world applications
- Empirical analysis on DevGPT dataset shows that LLM-generated code is often used for illustrating concepts or examples rather than production-ready code
- Further improvement is needed to enhance LLMs for seamless integration into software development practices
- Advancements like CodeGPT, CodeParrot, and Codex show potential to revolutionize software engineering tasks through human-AI collaboration
- Practical challenges and limitations need to be addressed for effective deployment of LLMs for code generation
- Continued research and development are essential to enhance LLM capabilities for real-world applications

## Summary

In the realm of Artificial Intelligence (AI), Large Language Models (LLMs) have emerged as powerful tools for code generation. These models have showcased impressive proficiency in various development scenarios such as code completion, source code mapping, and system maintenance. However, most evaluations of LLMs have been conducted in research settings, leaving a significant gap in understanding their effectiveness in real-world applications. To bridge this gap, our study delves into an empirical analysis of conversations within DevGPT - a dataset derived from developers' interactions with ChatGPT on platforms like GitHub. Our findings reveal that the current utilization of LLM-generated code is often limited to illustrating high-level concepts or providing examples in documentation rather than being deployed as production-ready code. This suggests the need for further improvement in enhancing LLMs for code generation before they can seamlessly integrate into modern software development practices. Recent advancements in LLMs such as CodeGPT, CodeParrot, and Codex underscore their potential to revolutionize software engineering tasks through collaborative efforts between humans and AI. As we navigate this paradigm shift towards more sophisticated AI-driven solutions, it becomes imperative to address the practical challenges and limitations associated with deploying LLMs for code generation effectively. In conclusion, our study sheds light on the current state of using LLMs for code generation and emphasizes the need for further research and development to enhance their capabilities for real-world applications. By leveraging insights from empirical analyses like ours, we can pave the way for more efficient and seamless integration of LLMs into modern software development workflows.