# Rethinking Code Review Workflows with LLM Assistance: An Empirical Study

**Authors:** Fannar Steinn Aðalsteinsson, Björn Borgar Magnússon, Mislav Milicevic, Adam Nirving Davidsson, Chih-Hong Cheng  
**arXiv:** [2505.16339](https://arxiv.org/abs/2505.16339)  
**Date:** 2025-05-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2505.16339v1/

## Key points

- Study conducted on rethinking code review workflows with LLM assistance
- Challenges and opportunities of incorporating Large Language Models (LLMs) in code review processes
- Key challenges in traditional code reviews: frequent context switching, insufficient contextual information
- Potential benefits of LLMs: automatic summarization of complex pull requests
- Concerns raised regarding false positives and trust issues when using LLMs
- Development of two prototype variations of the code review tool: one providing upfront LLM-generated reviews, the other allowing on-demand interaction
- Both variations employed a semantic search pipeline based on retrieval-augmented generation for gathering relevant contextual information
- Developers generally preferred AI-led reviews, but preferences were influenced by familiarity with the code base and severity of the pull request.

## Summary

, , , , 

In the study "Rethinking Code Review Workflows with LLM Assistance: An Empirical Study" conducted by Fannar Steinn Aðalsteinsson, Björn Borgar Magnússon, Mislav Milicevic, Adam Nirving Davidsson, and Chih-Hong Cheng, the authors delve into the challenges and opportunities presented by incorporating Large Language Models (LLMs) in code review processes. The research was carried out at WirelessCar Sweden AB and involved an exploratory field study of current code review practices along with a field experiment using two variations of an LLM-assisted code review tool. The field study revealed key challenges faced in traditional code reviews, such as frequent context switching and insufficient contextual information. It also highlighted potential benefits like automatic summarization of complex pull requests but also raised concerns regarding false positives and trust issues when utilizing LLMs. To address these challenges, the researchers developed two prototype variations of the code review tool: one providing LLM-generated reviews upfront and the other allowing on-demand interaction. Both variations employed a semantic search pipeline based on retrieval-augmented generation to gather relevant contextual information for the review process. In real-world settings, developers evaluated both prototype variations, with AI-led reviews being generally preferred. However, preferences were contingent on reviewers' familiarity with the code base and the severity of the pull request under consideration. This study sheds light on how integrating LLMs into code review workflows can enhance efficiency and effectiveness while also highlighting important considerations for successful implementation in practice.