# Examining Zero-Shot Vulnerability Repair with Large Language Models

**Authors:** Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, Brendan Dolan-Gavitt  
**arXiv:** [2112.02125](https://arxiv.org/abs/2112.02125)  
**Date:** 2021-12-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2112.02125v3/

## Key points

- Large language models (LLMs) can assist in repairing cybersecurity bugs in code
- OpenAI's Codex and AI21's Jurassic J-1 are examples of LLMs used for zero-shot vulnerability repair
- Challenges exist in designing prompts that effectively coax LLMs into generating repaired code due to semantic and syntactic variations in natural languages
- Comprehensive study involving commercially available LLMs, an open-source model, and a locally-trained model was conducted
- LLMs successfully repaired 100% of synthetically generated and hand-crafted scenarios
- Challenges identified when generating functionally correct code from historical real-world examples
- Emerging 'smart' code completion tools powered by LLMs have potential in addressing cybersecurity bugs introduced by human developers
- Further improvements needed to ensure generation of functionally accurate code

## Summary

In their research titled "Examining Zero-Shot Vulnerability Repair with Large Language Models," authors Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt explore the potential of large language models (LLMs) to assist in repairing cybersecurity bugs in code. They investigate the use of LLMs such as OpenAI's Codex and AI21's Jurassic J-1 for zero-shot vulnerability repair and analyze the challenges involved in designing prompts that can effectively coax LLMs into generating repaired versions of insecure code. The researchers highlight the difficulty in formulating prompts that convey key information accurately and comprehensively using natural language due to the semantic and syntactic variations present in natural languages. To evaluate the effectiveness of LLMs for vulnerability repair, they conduct a comprehensive study involving five commercially available black-box LLMs, an open-source model, and their own locally-trained model. The study encompasses a mix of synthetic scenarios, hand-crafted examples, and real-world security bug scenarios. The experiments demonstrate promising results as the collective performance of the LLMs successfully repairs 100% of synthetically generated and hand-crafted scenarios. However, when evaluating the models' performance on a corpus of historical real-world examples, they identify challenges in generating functionally correct code. Overall, this research sheds light on the potential role of emerging 'smart' code completion tools powered by LLMs in addressing cybersecurity bugs introduced by human developers. While there is promise in leveraging these tools for vulnerability repair, further improvements are necessary to ensure the generation of functionally accurate code.