Secrets of RLHF in Large Language Models Part I: PPO

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Cites this paper Not relevant

   Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment

   Tian-Hao Wu, Bang-Hua Zhu, Ruoyu Zhang et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 85 times · Not summarized yet

   Abstract

   Large Language Models (LLMs) can acquire extensive world knowledge through pre-training on large corpora. However, due to exposure to low-quality data, LLMs may exhibit harmful behavior without aligning with human values. The dominant approach for steering LLMs towards beneficial behavior involves Reinforcement Learning with Human Feedback (RLHF), with Proximal Policy Optimization (PPO) serving as the default RL optimizer. Despite its effectiveness, PPO has limitations when optimizing rewards trained from comparison-based loss. Primarily, PPO is not invariant to equivalent reward functions containing identical preference information due to the need to calibrate the reward scale. Additionally…

2. Very close Cites this paper Not relevant

   A Technical Survey of Reinforcement Learning Techniques for Large Language Models

   S. Srivastava, Vaneet Aggarwal · 2025 · ACM Transactions on Intelligent Systems and Technology · cited 29 times · Not summarized yet

   Abstract

   Reinforcement Learning (RL) has emerged as a transformative approach for aligning and enhancing Large Language Models (LLMs), addressing critical challenges in instruction following, ethical alignment, and reasoning capabilities. This survey offers a comprehensive foundation on the integration of RL with language models, highlighting prominent algorithms such as Proximal Policy Optimization (PPO), Q-Learning, and Actor-Critic methods. Additionally, it provides an extensive technical overview of RL techniques specifically tailored for LLMs, including foundational methods like Reinforcement Learning from Human Feedback (RLHF) and AI Feedback (RLAIF), as well as advanced strategies such as Dire…

3. Very close Not relevant

   Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms

   Xue-Rui Su, Yue Wang, Jinhua Zhu et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 8 times · Not summarized yet

   Abstract

   With the rapid development of Large Language Models (LLMs), numerous Reinforcement Learning from Human Feedback (RLHF) algorithms have been introduced to improve model safety and alignment with human preferences. These algorithms can be divided into two main frameworks based on whether they require an explicit reward (or value) function for training: actor-critic-based Proximal Policy Optimization (PPO) and alignment-based Direct Preference Optimization (DPO). The mismatch between DPO and PPO, such as DPO's use of a classification loss driven by human-preferred data, has raised confusion about whether DPO should be classified as a Reinforcement Learning (RL) algorithm. To address these ambig…

4. Very close Not relevant

   Teaching Large Language Models to Reason with Reinforcement Learning

   Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy et al. · 2024 · Machine Learning · preprint, not peer reviewed · cited 197 times · AI summary available

   Abstract

   Reinforcement Learning from Human Feedback (\textbf{RLHF}) has emerged as a dominant approach for aligning LLM outputs with human preferences. Inspired by the success of RLHF, we study the performance of multiple algorithms that learn from feedback (Expert Iteration, Proximal Policy Optimization (\textbf{PPO}), Return-Conditioned RL) on improving LLM reasoning capabilities. We investigate both sparse and dense rewards provided to the LLM both heuristically and via a learned reward model. We additionally start from multiple model sizes and initializations both with and without supervised fine-tuning (\textbf{SFT}) data. Overall, we find all algorithms perform comparably, with Expert Iteration…

5. Very close Not relevant

   Fine-Tuning Language Models with Advantage-Induced Policy Alignment

   Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri et al. · 2023 · Computation and Language · preprint, not peer reviewed · cited 52 times · AI summary available

   Abstract

   Reinforcement learning from human feedback (RLHF) has emerged as a reliable approach to aligning large language models (LLMs) to human preferences. Among the plethora of RLHF techniques, proximal policy optimization (PPO) is of the most widely used methods. Despite its popularity, however, PPO may suffer from mode collapse, instability, and poor sample efficiency. We show that these issues can be alleviated by a novel algorithm that we refer to as Advantage-Induced Policy Alignment (APA), which leverages a squared error loss function based on the estimated advantages. We demonstrate empirically that APA consistently outperforms PPO in language tasks by a large margin, when a separate reward …

6. Very close Not relevant

   A First-Order Logic-Based Alternative to Reward Models in RLHF

   Chun-Jin Jiang, Xin-Hua Zhu · 2025 · ICON · Not summarized yet

   Abstract

   Reinforcement Learning from Human Feedback (RLHF) plays a crucial role in aligning large language models (LLMs) with human values and preferences. However, the quality and stability of the trained reward model largely determine the final alignment performance. Existing approaches such as Proximal Policy Optimization (PPO) rely heavily on reward models to guide LLMs toward human-aligned behaviors. In this work, we propose a logic-similarity-based reward mechanism as an alternative to conventional reward modeling. Instead of relying on heuristic reward estimation, our method leverages formal logical consistency to steer model alignment with human preferences. Since real-world questions can be …

7. Very close Cites this paper Not relevant

   ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models

   Ziniu Li, Tian Xu, Yu-Shun Zhang et al. · 2023 · International Conference on Machine Learning · cited 221 times · Not summarized yet

   Abstract

   Reinforcement Learning from Human Feedback (RLHF) is key to aligning Large Language Models (LLMs), typically paired with the Proximal Policy Optimization (PPO) algorithm. While PPO is a powerful method designed for general reinforcement learning tasks, it is overly sophisticated for LLMs, leading to laborious hyper-parameter tuning and significant computation burdens. To make RLHF efficient, we present ReMax, which leverages 3 properties of RLHF: fast simulation, deterministic transitions, and trajectory-level rewards. These properties are not exploited in PPO, making it less suitable for RLHF. Building on the renowned REINFORCE algorithm, ReMax does not require training an additional value …

8. Very close Not relevant

   Reward-Robust RLHF in LLMs

   Yuzi Yan, Xingzhou Lou, J. Li et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 35 times · Not summarized yet

   Abstract

   As Large Language Models (LLMs) continue to progress toward more advanced forms of intelligence, Reinforcement Learning from Human Feedback (RLHF) is increasingly seen as a key pathway toward achieving Artificial General Intelligence (AGI). However, the reliance on reward-model-based (RM-based) alignment methods introduces significant challenges due to the inherent instability and imperfections of Reward Models (RMs), which can lead to critical issues such as reward hacking and misalignment with human intentions. In this paper, we introduce a reward-robust RLHF framework aimed at addressing these fundamental challenges, paving the way for more reliable and resilient learning in LLMs. Our app…

Prior work it builds on

1. Close Used in depth by this paper A method it uses Not relevant

   Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

   Yuntao Bai, Andy Jones, Kamal Ndousse et al. · 2022 · Computation and Language · preprint, not peer reviewed · cited 4,480 times · AI summary available

   Abstract

   We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence be…

2. Close Used in depth by this paper A method it uses Not relevant

   A General Language Assistant as a Laboratory for Alignment

   Amanda Askell, Yuntao Bai, Anna Chen et al. · 2021 · arXiv.org · preprint, not peer reviewed · cited 1,313 times · Not summarized yet

   Abstract

   Given the broad capabilities of large language models, it should be possible to work towards a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless. As an initial foray in this direction we study simple baseline techniques and evaluations, such as prompting. We find that the benefits from modest interventions increase with model size, generalize to a variety of alignment evaluations, and do not compromise the performance of large models. Next we investigate scaling trends for several training objectives relevant to alignment, comparing imitation learning, binary discrimination, and ranked preference modeling. We find that r…

3. Close Used in depth by this paper Not relevant

   Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO

   Logan Engstrom, Andrew Ilyas, Shibani Santurkar et al. · 2020 · arXiv.org · preprint, not peer reviewed · cited 365 times · Not summarized yet

   Abstract

   We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms: Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). Specifically, we investigate the consequences of "code-level optimizations:" algorithm augmentations found only in implementations or described as auxiliary details to the core algorithm. Seemingly of secondary importance, such optimizations turn out to have a major impact on agent behavior. Our results show that they (a) are responsible for most of PPO's gain in cumulative reward over TRPO, and (b) fundamentally change how RL methods function. These insights show the difficulty and import…

4. Related Used in depth by this paper A method it uses Not relevant

   LLaMA: Open and Efficient Foundation Language Models

   Hugo Touvron, Thibaut Lavril, Gautier Izacard et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 22,520 times · AI summary available

   Abstract

   We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.

5. Related Used in depth by this paper A method it uses Not relevant

   Training language models to follow instructions with human feedback

   Long Ouyang, Jeff Wu, Xu Jiang et al. · 2022 · Neural Information Processing Systems · cited 25,738 times · AI summary available

   Abstract

   Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outp…

Papers that cite it

1. Very close Not relevant

   Secrets of RLHF in Large Language Models Part II: Reward Modeling

   Binghai Wang, Rui Zheng, Lu Chen et al. · 2024 · Artificial Intelligence · preprint, not peer reviewed · cited 189 times · Not summarized yet

   Abstract

   Reinforcement Learning from Human Feedback (RLHF) has become a crucial technology for aligning language models with human values and intentions, enabling models to produce more helpful and harmless responses. Reward models are trained as proxies for human preferences to drive reinforcement learning optimization. While reward models are often considered central to achieving high performance, they face the following challenges in practical applications: (1) Incorrect and ambiguous preference pairs in the dataset may hinder the reward model from accurately capturing human intent. (2) Reward models trained on data from a specific distribution often struggle to generalize to examples outside that…

2. Close Not relevant

   VinePPO: Refining Credit Assignment in RL Training of LLMs

   Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance et al. · 2024 · International Conference on Machine Learning · cited 181 times · Not summarized yet

   Abstract

   Large language models (LLMs) are increasingly applied to complex reasoning tasks that require executing several complex steps before receiving any reward. Properly assigning credit to these steps is essential for enhancing model performance. Proximal Policy Optimization (PPO), a common reinforcement learning (RL) algorithm used for LLM finetuning, employs value networks to tackle credit assignment. However, recent approaches achieve strong results without it, raising questions about the efficacy of value networks in practice. In this work, we systematically evaluate the efficacy of value networks and reveal their significant shortcomings in reasoning-heavy LLM tasks, showing that they often …

3. Close Not relevant

   The Perfect Blend: Redefining RLHF with Mixture of Judges

   Tengyu Xu, Eryk Helenowski, Karthik Abinav Sankararaman et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 54 times · Not summarized yet

   Abstract

   Reinforcement learning from human feedback (RLHF) has become the leading approach for fine-tuning large language models (LLM). However, RLHF has limitations in multi-task learning (MTL) due to challenges of reward hacking and extreme multi-objective optimization (i.e., trade-off of multiple and/or sometimes conflicting objectives). Applying RLHF for MTL currently requires careful tuning of the weights for reward model and data combinations. This is often done via human intuition and does not generalize. In this work, we introduce a novel post-training paradigm which we called Constrained Generative Policy Optimization (CGPO). The core of CGPO is Mixture of Judges (MoJ) with cost-efficient co…

4. Close Not relevant

   Reward Shaping to Mitigate Reward Hacking in RLHF

   Jiayi Fu, Xuandong Zhao, Cheng-Yuan Yao et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 97 times · Not summarized yet

   Abstract

   Reinforcement learning from human feedback (RLHF) is widely used to align large language models (LLMs) with human preferences. However, RLHF remains vulnerable to \emph{reward hacking}, whereby a policy exploits imperfections in the reward function instead of learning the intended behavior, thereby undermining alignment. Although reward shaping can stabilize RLHF training and partially mitigate reward hacking, shaping methods and their underlying design principles have not been systematically investigated. To address this gap, we conduct a comprehensive study of prevalent reward-shaping techniques. Our analysis identifies two key design principles: (1) the reinforcement-learning reward shoul…

5. Very close Not relevant

   A framework for mitigating malicious RLHF feedback in LLM training using consensus based reward

   Zafaryab Haider, Md Hafizur Rahman, Vijay Devabhaktuni et al. · 2025 · Scientific Reports · cited 14 times · Not summarized yet

   Abstract

   Large Language models (LLMs) have demonstrated impressive capabilities in natural language processing and understanding. LLMs are being rapidly adopted in major industry sectors including mobile computing, healthcare, finance, government, and education driven by technology giants such as NVIDIA, OpenAI, Microsoft, Apple, Meta, Google, Broadcom, AMD, and IBM. However, due to the emerging nature of this technology, many security/privacy challenges remain unresolved that we must tackle before rolling out LLMs to critical applications (e.g. Healthcare, Legal). In this article, we focus on the Reinforcement Learning via Human Feedback (RLHF) process that is widely used for training LLMs giving th…