This paper presents an in-depth investigation into the capability of ChatGPT-3.5 and ChatGPT-4, two state-of-the-art Large Language Models (LLMs), for fault localisation in large-scale open-source programs. The study compares the performance of these LLMs with existing fault localisation techniques using the Defects4J dataset. The researchers also examine the stability and explanation of LLMs in fault localisation, as well as the impact of prompt engineering and code context length on effectiveness. The findings reveal that within a limited code context, ChatGPT-4 outperforms existing methods, achieving an average 46.9% higher accuracy compared to the state-of-the-art baseline SmartFL. However, when the code context expands to the class level, ChatGPT models become less effective than existing methods overall. Additionally, ChatGPT's explainability is found to be unsatisfactory with only approximately 30% accuracy. These limitations highlight the need for further research to fully harness the potential of LLMs like ChatGPT for practical fault localisation applications.
- ChatGPT-3.5 and ChatGPT-4 are investigated for fault localisation in large-scale open-source programs
- Performance of these models is compared to existing fault localisation techniques using the Defects4J dataset
- Stability and explanation of LLMs in fault localisation are examined
- Impact of prompt engineering and code context length on effectiveness is analyzed
- ChatGPT-4 outperforms existing methods within limited code context, achieving 46.9% higher accuracy than SmartFL baseline
- However, when code context expands to class level, ChatGPT models become less effective overall
- ChatGPT's explainability is found to be unsatisfactory with only approximately 30% accuracy
- Further research is needed to fully harness the potential of LLMs like ChatGPT for practical fault localisation applications.
Researchers have been studying two computer programs called ChatGPT-3.5 and ChatGPT-4 to find mistakes in big computer programs that are freely available. They compared how well these programs work to other methods of finding mistakes using a special set of data called Defects4J. They also looked at how stable and understandable the programs are when finding mistakes. They found that ChatGPT-4 is better than other methods when only a small part of the program is used, but not as good when they use a bigger part of the program. The researchers also found that ChatGPT's explanations for mistakes are not very accurate. They think more research is needed to make these programs even better at finding mistakes in real-life situations."
Definitions1. Fault localisation: Finding and identifying mistakes or errors in computer programs.
2. Dataset: A collection of data or information that is used for analysis or study.
3. Prompt engineering: The process of designing specific instructions or questions to guide an artificial intelligence model's response.
4. Code context: The surrounding code or programming instructions that provide information about how a specific part of a program works.
5. Accuracy: How correct or precise something is compared to the truth or desired outcome.
6. Explainability: The ability to understand and explain why something happens or works in a certain way.
7. Potential: The possibility for something to happen or be developed in the future
Exploring the Capability of ChatGPT-3.5 and ChatGPT-4 for Fault Localisation in Large-Scale Open Source Programs
Fault localisation is a process used to identify the source of errors or bugs in software programs. It is an important task for developers, as it helps them quickly locate and fix issues with their code. Recently, researchers have been exploring the potential of large language models (LLMs) such as ChatGPT-3.5 and ChatGPT-4 for fault localisation applications. In this article, we will discuss a research paper that investigates these LLMs’ capability for fault localisation in large open source programs using the Defects4J dataset.
Background
Fault localisation techniques are used to pinpoint which parts of a program contain faults or bugs by comparing its expected output with actual output from running tests on it. This can be done manually by experienced developers or automatically using existing methods such as Tarantula, Ochiai, Jaccard, etc., which rely on static analysis to compute similarity scores between suspicious statements and failing test cases. However, these methods often fail when dealing with complex programs due to their limited ability to capture contextual information about the codebase.
Recently, researchers have been exploring how LLMs like ChatGPT can be applied to fault localisation tasks due to their capacity for capturing long-term dependencies between tokens within a given context. The two most recent versions of this model are ChatGPT-3.5 and ChatGPT-4; both are transformer architectures that use masked language modelling (MLM) pre-training objectives combined with prompt engineering strategies tailored specifically towards programming tasks such as debugging and refactoring code snippets.
Research Methodology
The study compares the performance of these LLMs with existing fault localisation techniques using the Defects4J dataset—a benchmark collection containing real faults from open source Java projects including Apache Commons Math 3 (ACM), Apache Commons Lang 3 (ACL), Eclipse JDT Core (ECJ), Google Guava Libraries (GGUAVA), Mockito 1 (MOCKITO1) and PMD 5 (PMD). To evaluate effectiveness, they measure precision at top k ranks—the percentage of correctly identified faulty statements among those ranked highest according to each technique’s similarity score—and mean average precision at top k ranks—the average precision over all test cases at different rank thresholds up until k=20 . Additionally, they examine explainability by calculating accuracy scores based on whether or not LLMs correctly identify faulty lines when provided only class level contexts instead of full method bodies where possible explanations could be found more easily than without them . Lastly, they investigate stability through experiments measuring how well models perform consistently across different datasets after being trained on one particular project .
Findings
The findings reveal that within a limited code context consisting only of method bodies ,ChatGPT - 4 outperforms existing methods , achieving an average 46 . 9 % higher accuracy compared to SmartFL —the state -of -the -art baseline technique . However , when expanding contexts up to class levels ,Chat G PT models become less effective overall than existing methods . Additionally , explainability was found unsatisfactory ; even though providing additional context improved results slightly , accuracy was still around 30 % lower than other techniques tested . These limitations highlight the need for further research into harnessing LLMs like Chat G PT effectively for practical applications related to fault localization .
Conclusion
This research paper provides an in depth investigation into the capability of two state -of -the art Large Language Models —Chat G PT - 3 . 5 and 4 —for fault localization in large scale open source programs using Defects 4 J dataset . The findings show that while within limited code contexts ,Chat G PT performs better than existing approaches ; however its effectiveness decreases significantly when expanded up until class levels along with unsatisfactory explainability rates highlighting need for further studies into harnessing its potential fully for practical applications related to fault localization
