# How to Refactor this Code? An Exploratory Study on Developer-ChatGPT Refactoring Conversations

**Authors:** Eman Abdullah AlOmar, Anushkrishna Venkatakrishnan, Mohamed Wiem Mkaouer, Christian D. Newman, Ali Ouni  
**arXiv:** [2402.06013](https://arxiv.org/abs/2402.06013)  
**Date:** 2024-02-08  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2402.06013v1/

## Key points

- Study focuses on interactions between developers and ChatGPT in code refactoring context
- Developers often provide code fragments and textual descriptions for refactoring needs
- ChatGPT offers informative suggestions but has limited understanding of broader codebase context
- Challenges can arise from missing dependencies, codependencies, compiler errors, and test failures
- Quality of prompts influences ChatGPT's performance; high-quality prompts yield effective responses
- Model performs well for non-urgent issues like code style improvements or adding comments
- Issues arise when ChatGPT misunderstands or gets confused by reported code fragments or incomplete inputs
- Different learning settings observed: zero-shot learning vs. few-shot learning during refactoring conversations
- Manual inspection of GitHub repositories reveals 43 refactoring documentation patterns in developer-ChatGPT conversations
- Findings contribute to understanding dynamics between developers and AI models in code refactoring, aiming to enhance model capabilities and software engineering best practices

## Summary

In this study, we examine the interactions between developers and ChatGPT in the context of code refactoring. Our goal is to gain insights into how developers articulate their refactoring needs and how ChatGPT addresses those needs. Through our analysis of 17,913 ChatGPT prompts and responses, we have discovered that developers often copy and paste code fragments that require refactoring along with textual descriptions of their desired changes. While ChatGPT provides informative suggestions for refactorings, its understanding of the broader context of the codebase is limited. This can lead to potential issues such as missing dependencies and codependencies. Additionally, developers may encounter challenges with suggested code changes including compiler errors and test failures resulting from ChatGPT-provided solutions. Our research also highlights the importance of high-quality prompts in eliciting effective responses from ChatGPT. We have found that the model's performance is closely tied to the quality of its training data. For non-urgent issues like code style improvements or adding comments, insightful suggestions are offered by ChatGPT. However, complexities arise when ChatGPT misunderstands or becomes confused by reported code fragments or incomplete inputs. By identifying common patterns and challenges in these conversations between developers and AI models like ChatGPT, researchers and developers can work towards enhancing its capabilities to better meet developer needs. Furthermore, our exploration has uncovered variability in learning settings during refactoring conversations between developers and ChatGPT. Some developers rely on zero-shot learning by using the model's generative ability to propose fixes for unseen data based on prior training. Others opt for few-shot learning by providing code fragments and checking for design antipatterns. Through manual inspection of GitHub repositories to identify refactoring documentation patterns represented as keywords or phrases in developer-ChatGPT conversations, we have compiled a list of 43 such patterns. Overall, our findings shed light on the dynamics between developers and AI models like ChatGPT in the realm of code refactoring. By understanding these interactions more deeply, we can pave the way for improving model performance, tool development, and best practices in software engineering.