# Do you still need a manual smart contract audit?

**Authors:** Isaac David, Liyi Zhou, Kaihua Qin, Dawn Song, Lorenzo Cavallaro, Arthur Gervais  
**arXiv:** [2306.12338](https://arxiv.org/abs/2306.12338)  
**Date:** 2023-06-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.12338v1/

## Key points

- Isaac David, Liyi Zhou, Kaihua Qin, Dawn Song, Lorenzo Cavallaro, and Arthur Gervais investigate the feasibility of using large language models (LLMs) for security audits of smart contracts.
- The researchers evaluate the performance and accuracy of LLMs using a benchmark dataset of 52 compromised DeFi smart contracts.
- GPT-4 and Claude models correctly identify vulnerability types in 40% of cases but have a high false positive rate.
- LLMs outperform a random model by 20% in terms of F1-score.
- Mutation testing on five newly developed secure smart contracts reveals a best-case true positive rate of 78.7% for the GPT-4-32k model.
- The models are evaluated using binary classification tasks and non-binary prompts, considering model temperature variations and context length.
- Manual auditors' involvement remains crucial due to the high false positive rate exhibited by LLMs.
- This research lays the groundwork for a more efficient and cost-effective approach to conducting smart contract security audits.

## Summary

In their research, Isaac David, Liyi Zhou, Kaihua Qin, Dawn Song, Lorenzo Cavallaro and Arthur Gervais investigate the feasibility of using large language models (LLMs) for conducting security audits of smart contracts. They aim to optimize prompt engineering for enhanced security analysis and evaluate the performance and accuracy of LLMs using a benchmark dataset consisting of 52 previously compromised Decentralized Finance (DeFi) smart contracts. The researchers find that when vulnerable contracts are analyzed, both the GPT-4 and Claude models correctly identify the vulnerability type in 40% of cases. However, these models also exhibit a high false positive rate, indicating the need for continued involvement from manual auditors. Despite this limitation, the LLMs tested outperform a random model by 20% in terms of F1-score. To ensure the integrity of their study, mutation testing is conducted on five newly developed supposedly secure smart contracts. The researchers manually insert two and 15 vulnerabilities into each contract. This testing reveals an impressive best-case true positive rate of 78.7% for the GPT-4-32k model. The models are evaluated using both binary classification tasks to determine contract vulnerability and non-binary prompts as well as examining the influence of model temperature variations and context length on LLM performance. Overall, this research highlights the potential benefits of employing LLMs in security audits but emphasizes that manual auditors' involvement remains crucial due to the high false positive rate exhibited by these models. Although there is room for further enhancements, this work lays the groundwork for a more efficient and cost-effective approach to conducting smart contract security audits.