# Stabilizing Reinforcement Learning with LLMs: Formulation and Practices

**Authors:** Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, An Yang, Jingren Zhou, Junyang Lin  
**arXiv:** [2512.01374](https://arxiv.org/abs/2512.01374)  
**Date:** 2025-12-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2512.01374v1/

## Key points

- Authors propose a novel formulation for reinforcement learning (RL) using large language models
- Surrogate token-level objective in policy gradient methods like REINFORCE optimizes true sequence-level reward
- Importance of techniques such as importance sampling correction, clipping, and Routing Replay for stabilizing RL training highlighted
- On-policy training with basic policy gradient algorithm and importance sampling correction yields highest training stability
- Off-policy updates to accelerate convergence require combining clipping and Routing Replay to mitigate instability from policy staleness
- Prolonged optimization leads to comparable final performance regardless of cold-start initialization

## Summary

In their paper titled "Stabilizing Reinforcement Learning with LLMs: Formulation and Practices," authors Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, An Yang, Jingren Zhou, and Junyang Lin propose a novel formulation for reinforcement learning (RL) using large language models. They explore how the true sequence-level reward can be optimized through a surrogate token-level objective in policy gradient methods like REINFORCE. By conducting a first-order approximation analysis, the authors demonstrate that this surrogate objective becomes increasingly valid when minimizing both training-inference discrepancy and policy staleness. The study sheds light on the importance of various techniques in stabilizing RL training including importance sampling correction, clipping and Routing Replay for Mixture-of-Experts (MoE) models. Through extensive experiments involving a 30B MoE model and hundreds of thousands of GPU hours,the authors find that on-policy training with the basic policy gradient algorithm and importance sampling correction yields the highest training stability. They also highlight that introducing off-policy updates to accelerate convergence requires combining clipping and Routing Replay to mitigate instability stemming from policy staleness. Once training is stabilized using these techniques,prolonged optimization consistently leads to comparable final performance regardless of cold-start initialization. The authors hope that by sharing their insights and developed strategies for stable RL training in this paper will facilitate future research in this area.