# DevEval: Evaluating Code Generation in Practical Software Projects

**Authors:** Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Zhi Jin, Hao Zhu, Huanyu Liu, Kaibo Liu, Lecheng Wang, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yihong Dong, Yuqi Zhu, Bin Gu, Mengfei Yang  
**arXiv:** [2401.06401](https://arxiv.org/abs/2401.06401)  
**Date:** 2024-01-12  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2401.06401v1/

## Key points

- The paper addresses the challenge of evaluating Large Language Models (LLMs) in code generation
- Many proposed benchmarks fail to align with practical software projects
- The authors propose a new benchmark called DevEval that aligns with developers' experiences in practical projects
- DevEval consists of 2,690 samples from 119 real-world projects across 10 domains
- It incorporates realistic program distributions, sufficient dependencies, and large-scale project contexts
- Five popular LLMs are evaluated on the DevEval benchmark
- The highest Pass@1 score achieved is only 42 in their experiments
- The paper discusses challenges faced in code generation for practical projects and highlights future directions for improvement
- Open-sourcing DevEval is emphasized to facilitate further development in code generation for practical software projects
- The paper presents a comprehensive evaluation framework for LLMs in code generation tasks based on real-world project data

## Summary

In their paper "DevEval: Evaluating Code Generation in Practical Software Projects," Jia Li et al. address the challenge of evaluating Large Language Models (LLMs) in code generation. Many proposed benchmarks fail to align with practical software projects due to unrealistic program distributions, insufficient dependencies, and small-scale project contexts. To overcome these limitations, the authors propose a new benchmark called DevEval that is specifically designed to align with developers' experiences in practical projects. <br>
<br>
DevEval is constructed through a rigorous pipeline and consists of 2,690 samples from 119 real-world projects across 10 domains. It incorporates realistic program distributions, sufficient dependencies, and large-scale project contexts unlike previous benchmarks. The authors evaluate five popular LLMs including gpt-4, gpt-3.5-turbo, CodeLLaMa, and StarCoder on the DevEval benchmark to assess their actual abilities in code generation. The results show that even the highest Pass@1 score achieved by gpt-3.5-turbo is only 42 in their experiments.<br>
<br>
The paper also discusses the challenges faced in code generation for practical projects and highlights future directions for improvement. The authors emphasize the importance of open-sourcing DevEval to facilitate further development in code generation for practical software projects.<br>
<br>
Overall, this paper presents a comprehensive evaluation framework for LLMs in code generation tasks based on real-world project data. The proposed benchmark provides valuable insights into the actual capabilities of popular LLMs and sheds light on areas that require further research and improvement.