# Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT

**Authors:** Burak Yetiştiren, Işık Özsoy, Miray Ayerdem, Eray Tüzün  
**arXiv:** [2304.10778](https://arxiv.org/abs/2304.10778)  
**Date:** 2023-04-21  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2304.10778v1/

## Key points

- The study evaluates code quality of AI-assisted code generation tools: GitHub Copilot, Amazon CodeWhisperer, and ChatGPT.
- Tools generate code from natural language prompts or partial code inputs.
- Objective is to compare performance in terms of various code quality metrics: Code Validity, Code Correctness, Code Security, Code Reliability, and Code Maintainability.
- Researchers used benchmark HumanEval Dataset to assess capabilities of tools.
- Results show correct code generation rates: ChatGPT 65.2%, GitHub Copilot 46.3%, Amazon CodeWhisperer 31.1%.
- Newer versions of GitHub Copilot and Amazon CodeWhisperer showed improvement rates in generating correct code.
- Average technical debt (considering code smells): ChatGPT 8.9 minutes, GitHub Copilot 9.1 minutes, Amazon CodeWhisperer 5.6 minutes.
- Study highlights strengths and weaknesses of these popular tools for practitioners in software development.
- Comparing performance metrics can help practitioners select optimal tool for specific tasks in software engineering projects.

## Summary

The study conducted by Burak Yetiştiren et al. evaluates the code quality of AI-assisted code generation tools, specifically focusing on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT. These tools have gained popularity in software engineering for their ability to generate code from natural language prompts or partial code inputs. The objective of the study is to compare the performance of these tools in terms of various code quality metrics such as Code Validity, Code Correctness, Code Security, Code Reliability, and Code Maintainability. To assess the capabilities of these prominent code generation tools, the researchers utilized the benchmark HumanEval Dataset. The generated code was then evaluated based on the proposed code quality metrics to determine their strengths and shortcomings. The results of the analysis indicate that the latest versions of ChatGPT, GitHub Copilot, and Amazon CodeWhisperer generate correct code 65.2%, 46.3%, and 31.1% of the time respectively. Furthermore, it was observed that newer versions of GitHub Copilot and Amazon CodeWhisperer showed improvement rates of 18% and 7% respectively in generating correct code. The average technical debt, considering code smells, was found to be 8.9 minutes for ChatGPT, 9.1 minutes for GitHub Copilot, and 5.6 minutes for Amazon CodeWhisperer. In conclusion, this study highlights both the strengths and weaknesses of these popular code generation tools providing valuable insights for practitioners in software development. By comparing these generators based on their performance metrics, practitioners can make informed decisions when selecting an optimal tool for specific tasks thereby enhancing their decision-making process in software engineering projects.