# Teaching Large Language Models to Reason with Reinforcement Learning

**Authors:** Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, Roberta Raileanu  
**arXiv:** [2403.04642](https://arxiv.org/abs/2403.04642)  
**Date:** 2024-03-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.04642v1/

## Key points

- Researchers explore effectiveness of reinforcement learning algorithms in improving reasoning capabilities of Large Language Models (LLMs)
- Investigate algorithms like Expert Iteration, Proximal Policy Optimization (PPO), and Return-Conditioned RL
- Analyze impact of sparse and dense rewards provided to LLM through heuristic methods or learned reward model
- Consider various model sizes and initializations with and without supervised fine-tuning (SFT) data
- Findings show all algorithms perform comparably, with Expert Iteration showing superior performance in most cases
- Sample complexity of Expert Iteration similar to PPO, requiring around $10^6$ samples to converge from pretrained checkpoint
- Trade-off between maj@1 and pass@96 metric performance during SFT training; RL training improves both metrics simultaneously
- RL fine-tuning less prone to overfitting compared to static SFT fine-tuning due to exploration process generating diverse solution paths
- RLHF results in better generalization than SFT on summarization and instruction following tasks
- PPO and Expert Iteration demonstrate nearly 10% improvement in pass@96 over continued SFT but smaller improvements over light SFT

## Summary

In the study "Teaching Large Language Models to Reason with Reinforcement Learning," researchers Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu explore the effectiveness of reinforcement learning algorithms in improving the reasoning capabilities of Large Language Models (LLMs). The research is motivated by the success of Reinforcement Learning from Human Feedback (RLHF) in aligning LLM outputs with human preferences. The researchers investigate multiple algorithms that learn from feedback, including Expert Iteration, Proximal Policy Optimization (PPO), and Return-Conditioned RL. They analyze the impact of sparse and dense rewards provided to the LLM through heuristic methods or a learned reward model. Additionally, they consider various model sizes and initializations with and without supervised fine-tuning (SFT) data. The findings reveal that all algorithms perform comparably, with Expert Iteration showing superior performance in most cases. Surprisingly, the sample complexity of Expert Iteration is similar to that of PPO, requiring around $10^6$ samples to converge from a pretrained checkpoint. The researchers attribute this phenomenon to the limited exploration beyond solutions already produced by SFT models during RL training. Moreover,the study highlights a trade-off between maj@1 and pass@96 metric performance during SFT training. Interestingly, RL training improves both metrics simultaneously. The researchers suggest that RL fine-tuning is less prone to overfitting compared to static SFT fine-tuning due to its exploration process generating diverse solution paths. Furthermore, recent work has shown that RLHF results in better generalization than SFT on summarization and instruction following tasks. Both PPO and Expert Iteration demonstrate nearly 10% improvement in pass@96 over continued SFT but show smaller improvements over light SFT. In conclusion, the study sheds light on the benefits of reinforcement learning algorithms for enhancing LLM reasoning capabilities. The findings have implications for RLHF applications and suggest a promising future role for RL in LLM fine-tuning.