# Can OpenAI Codex and Other Large Language Models Help Us Fix Security Bugs?

**Authors:** Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, Brendan Dolan-Gavitt  
**arXiv:** [2112.02125](https://arxiv.org/abs/2112.02125)  
**Date:** 2021-12-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2112.02125v1/

## Key points

- Study title: "Can OpenAI Codex and Other Large Language Models Help Us Fix Security Bugs?"
- Authors: Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, Brendan Dolan-Gavitt
- Purpose: Explore potential of large language models (LLMs) in repairing cybersecurity weaknesses in code
- Methodology: Design prompts to guide LLMs in generating repaired versions of insecure code
- Challenges: Crafting accurate prompts using natural language
- Evaluation: Comprehensive study involving four commercially available black-box LLMs and a locally-trained model
- LLMs repaired 100% of synthetically generated and hand-crafted scenarios
- Addressed 58% of vulnerabilities in historical bugs within real-world open-source projects
- Implications: Potential for enhancing cybersecurity practices in software development using advanced language models

## Summary

In their study titled "Can OpenAI Codex and Other Large Language Models Help Us Fix Security Bugs? ", authors Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt explore the potential of large language models (LLMs) in repairing cybersecurity weaknesses in code. The researchers investigate the effectiveness of these 'smart' code completion tools for zero-shot vulnerability repair by designing prompts that guide LLMs to generate repaired versions of insecure code. They highlight the challenges in crafting prompts that accurately convey key information using natural language. Through a comprehensive study involving four commercially available black-box LLMs and a locally-trained model, the team evaluates the performance of these models on various scenarios including synthetic, hand-crafted, and real-world security bugs. Their experiments reveal promising results, showing that collectively LLMs were able to repair 100% of synthetically generated and hand-crafted scenarios as well as address 58% of vulnerabilities identified in historical bugs within real-world open-source projects. This research highlights the potential of leveraging advanced language models for enhancing cybersecurity practices in software development. By demonstrating the capability of LLMs to effectively identify and rectify security vulnerabilities in code, this study paves the way for further exploration into utilizing artificial intelligence technologies for bolstering cyber defense mechanisms within programming environments.