# How ChatGPT is Solving Vulnerability Management Problem

**Authors:** Peiyu Liu, Junming Liu, Lirong Fu, Kangjie Lu, Yifan Xia, Xuhong Zhang, Wenzhi Chen, Haiqin Weng, Shouling Ji, Wenhai Wang  
**arXiv:** [2311.06530](https://arxiv.org/abs/2311.06530)  
**Date:** 2023-11-11  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2311.06530v1/

## Key points

- Significant interest in the code analysis domain regarding ChatGPT
- Previous studies demonstrate ChatGPT's capability in foundational code analysis tasks
- Evaluation of ChatGPT's performance in vulnerability management tasks using a large-scale dataset
- Promising potential for leveraging ChatGPT in vulnerability management, particularly in generating titles for software bug reports
- Difficulties faced by ChatGPT and areas for future research identified
- Challenges with providing random demonstration examples and guiding ChatGPT to focus on relevant information
- Study expands understanding of how ChatGPT can contribute to vulnerability management and provides insights for future research directions.

## Summary

Recently, there has been significant interest in the code analysis domain regarding ChatGPT. Previous studies have demonstrated that ChatGPT is capable of processing foundational code analysis tasks, such as generating abstract syntax trees, indicating its potential for understanding code syntax and static behaviors. To further explore ChatGPT's capabilities in vulnerability management tasks, the authors of this paper evaluate its performance across six tasks related to the complete vulnerability management process. They utilize a large-scale dataset consisting of 78,445 samples and compare ChatGPT against state-of-the-art approaches. The results of their experiments indicate promising potential for leveraging ChatGPT to assist in vulnerability management. Notably, ChatGPT demonstrates proficiency in tasks like generating titles for software bug reports. However, the study also uncovers difficulties faced by ChatGPT and highlights areas for future research. For instance, directly providing random demonstration examples in the prompt does not consistently guarantee good performance in vulnerability management; instead a self-heuristic approach that extracts expertise from demonstration examples themselves and integrates it into the prompt shows promise. Furthermore, effectively guiding ChatGPT to focus on relevant information rather than irrelevant content remains an open problem. Overall, this study expands our understanding of how ChatGPT can contribute to vulnerability management by exploring its capabilities across multiple tasks and shedding light on both its strengths and limitations. The findings provide valuable insights for future research directions aimed at improving the performance and usability of AI models like ChatGPT in practical code analysis scenarios.