# Teaching Large Language Models to Self-Debug

**Authors:** Xinyun Chen, Maxwell Lin, Nathanael Schärli, Denny Zhou  
**arXiv:** [2304.05128](https://arxiv.org/abs/2304.05128)  
**Date:** 2023-04-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.05128v1/

## Key points

- Large language models (LLMs) have impressive performance in code generation, but generating correct solutions for complex programming tasks can be challenging.
- Prior works have designed program repair approaches to improve code generation performance.
- The authors propose a novel approach called Self-Debugging which teaches a large language model to debug its predicted program via few-shot demonstrations.
- The approach enables the model to perform rubber duck debugging without any feedback on code correctness or error messages.
- Self-Debugging achieves state-of-the-art performance on several code generation benchmarks including text-to-SQL generation, C++-to-Python translation and text-to-Python generation.
- On the Spider benchmark where there are no unit tests to verify prediction correctness, Self-Debugging with code explanation consistently improves the baseline by 2–3% and improves prediction accuracy on problems of the hardest label by 9%.
- On TransCoder and MBPP where unit tests are available, Self-Debugging improves baseline accuracy by up to 12%.
- By leveraging feedback messages and reusing failed predictions, Self Debugging notably improves sample efficiency and can match or outperform baseline models that generate more than 10x candidate programs.
- Future work includes improving the model's ability to conduct all these steps through better code explanation ability leading to better debugging performance.

## Summary

Large language models (LLMs) have shown impressive performance in code generation, but generating the correct solution for complex programming tasks can be challenging. To address this issue, prior works have designed program repair approaches to improve code generation performance. In this work, the authors propose a novel approach called Self-Debugging which teaches a large language model to debug its predicted program via few-shot demonstrations. The approach enables the model to perform rubber duck debugging without any feedback on code correctness or error messages. The model identifies its mistakes by explaining the generated code in natural language. The authors demonstrate that Self-Debugging achieves state-of-the-art performance on several code generation benchmarks including text-to-SQL generation, C++-to-Python translation and text-to-Python generation. On the Spider benchmark where there are no unit tests to verify prediction correctness, Self-Debugging with code explanation consistently improves the baseline by 2–3% and improves prediction accuracy on problems of the hardest label by 9%. On TransCoder and MBPP where unit tests are available, Self-Debugging improves baseline accuracy by up to 12%. By leveraging feedback messages and reusing failed predictions, Self Debugging notably improves sample efficiency and can match or outperform baseline models that generate more than 10x candidate programs. The authors highlight the promise of improving coding performance of large language models by teaching them to iteratively debug their own predictions instead of requiring them to generate correct code from scratch. SELF DEBUGGING instructs the model to understand the code identify errors and follow error messages to fix bugs. Future work includes improving the model's ability to conduct all these steps through better code explanation ability leading to better debugging performance. One direction is instructing models to describe high level semantic meaning of code along with implementation details in explanations while another is including additional debugging information in feedback such as potential bug descriptions. Preliminary results suggest that model generated feedback messages on semantic errors do not provide additional benefits on top of line–by–line code explanation and future work can explore techniques to predict more informative error messages.