# GitHub Copilot AI pair programmer: Asset or Liability?

**Authors:** Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, Zhen Ming,  Jiang  
**arXiv:** [2206.15331](https://arxiv.org/abs/2206.15331)  
**Date:** 2022-06-30  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2206.15331v1/

## Key points

- Open AI and Microsoft proposed a Deep Learning (DL) based solution called Copilot for automatic program synthesis in software engineering.
- More empirical evaluations are necessary to understand how developers can effectively benefit from Copilot.
- The study assessed the capabilities of Copilot in two different programming tasks: generating correct and efficient solutions for fundamental algorithmic problems, and comparing its proposed solutions with those generated by human programmers on a set of programming tasks.
- Copilot is capable of providing solutions for almost all fundamental algorithmic problems, but some solutions were buggy and non-reproducible. Additionally, it had difficulties combining multiple methods to generate a solution.
- While the correct ratio of human solutions was greater than Copilot's correct ratio, buggy solutions generated by Copilot required less effort to repair.
- Further research is necessary to improve the accuracy and effectiveness of Copilot as an industrial product.

## Summary

The field of software engineering has long sought an automatic program synthesis solution, and recently, Open AI and Microsoft proposed a promising Deep Learning (DL) based solution called Copilot. Studies have evaluated the correctness of Copilot solutions and reported issues; however, more empirical evaluations are necessary to understand how developers can effectively benefit from it. This paper presents a study that assesses the capabilities of Copilot in two different programming tasks: generating correct and efficient solutions for fundamental algorithmic problems, and comparing its proposed solutions with those generated by human programmers on a set of programming tasks. For the first task, the researchers assessed the performance and functionality of Copilot in solving selected fundamental problems in computer science such as sorting and implementing basic data structures. The results showed that Copilot is capable of providing solutions for almost all fundamental algorithmic problems; however, some solutions were buggy and non-reproducible. Additionally, Copilot had difficulties combining multiple methods to generate a solution. In the second task, the researchers used a dataset of programming problems with human-provided solutions to compare Copilot's proposed solutions with those generated by humans. The results showed that while the correct ratio of human solutions was greater than Copilot's correct ratio, buggy solutions generated by Copilot required less effort to repair. Overall, while Copilot shows limitations as an assistant for developers especially in advanced programming tasks as highlighted in this study and previous ones, it can generate preliminary solutions for basic programming tasks. Therefore, further research is necessary to improve its accuracy and effectiveness as an industrial product.