# Large Language Models in Fault Localisation

**Authors:** Yonghao Wu, Zheng Li, Jie M. Zhang, Mike Papadakis, Mark Harman, Yong Liu  
**arXiv:** [2308.15276](https://arxiv.org/abs/2308.15276)  
**Date:** 2023-08-29  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2308.15276v1/

## Key points

- ChatGPT-3.5 and ChatGPT-4 are investigated for fault localisation in large-scale open-source programs
- Performance of these models is compared to existing fault localisation techniques using the Defects4J dataset
- Stability and explanation of LLMs in fault localisation are examined
- Impact of prompt engineering and code context length on effectiveness is analyzed
- ChatGPT-4 outperforms existing methods within limited code context, achieving 46.9% higher accuracy than SmartFL baseline
- However, when code context expands to class level, ChatGPT models become less effective overall
- ChatGPT's explainability is found to be unsatisfactory with only approximately 30% accuracy
- Further research is needed to fully harness the potential of LLMs like ChatGPT for practical fault localisation applications.

## Summary

This paper presents an in-depth investigation into the capability of ChatGPT-3.5 and ChatGPT-4, two state-of-the-art Large Language Models (LLMs), for fault localisation in large-scale open-source programs. The study compares the performance of these LLMs with existing fault localisation techniques using the Defects4J dataset. The researchers also examine the stability and explanation of LLMs in fault localisation, as well as the impact of prompt engineering and code context length on effectiveness. The findings reveal that within a limited code context, ChatGPT-4 outperforms existing methods, achieving an average 46.9% higher accuracy compared to the state-of-the-art baseline SmartFL. However, when the code context expands to the class level, ChatGPT models become less effective than existing methods overall. Additionally, ChatGPT's explainability is found to be unsatisfactory with only approximately 30% accuracy. These limitations highlight the need for further research to fully harness the potential of LLMs like ChatGPT for practical fault localisation applications.