# Training Language Models to Self-Correct via Reinforcement Learning

**Authors:** Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, Aleksandra Faust  
**arXiv:** [2409.12917](https://arxiv.org/abs/2409.12917)  
**Date:** 2024-09-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2409.12917v1/

## Key points

- Self-correction is a crucial capability lacking in modern Large Language Models (LLMs)
- Existing methods for training self-correction often require multiple models or supervision from a more advanced model
- SCoRe is a multi-turn online reinforcement learning approach that enhances LLMs' self-correction ability using self-generated data
- Traditional supervised fine-tuning on correction traces generated offline was insufficient for effective self-correction behavior
- SCoRe trains on the model's own distribution of self-generated correction traces and implements appropriate regularization to guide the learning process
- SCoRe involves an initial phase of RL on a base model followed by reward bonuses to amplify self-correction during training
- When applied to Gemini 1.0 Pro and 1.5 Flash models, SCoRe demonstrated significant improvements in self-correction performance on MATH and HumanEval benchmarks
- Reinforcement learning can effectively train language models in self-correction tasks without external supervision, achieving state-of-the-art results

## Summary

In the realm of Large Language Models (LLMs), self-correction is a crucial capability that has unfortunately been lacking in modern LLMs. Existing methods for training self-correction often require multiple models or supervision from a more advanced model. To address this issue, a team of researchers developed SCoRe, a multi-turn online reinforcement learning (RL) approach that enhances an LLM's self-correction ability using self-generated data. The researchers discovered that traditional supervised fine-tuning (SFT) on correction traces generated offline by the model was insufficient for instilling effective self-correction behavior. Training through SFT resulted in either a mismatch between the training data and the model's responses or favored specific correction behaviors that were not effective during testing. SCoRe overcomes these challenges by training on the model's own distribution of self-generated correction traces and implementing appropriate regularization to guide the learning process towards developing a successful self-correction strategy at test time, rather than merely fitting high-reward responses. SCoRe involves running an initial phase of RL on a base model to establish a policy initialization less prone to collapse, followed by utilizing reward bonuses to amplify self-correction during training. When applied to Gemini 1.0 Pro and 1.5 Flash models, SCoRe demonstrated significant improvements in self-correction performance, enhancing the base models' capabilities by 15.6% and 9.1% respectively on MATH and HumanEval benchmarks. Overall, this innovative approach showcases how reinforcement learning can be leveraged to train language models effectively in self-correction tasks without relying on external supervision or multiple models, ultimately achieving state-of-the-art results in enhancing LLMs' ability to correct themselves accurately and efficiently.