# Secrets of RLHF in Large Language Models Part I: PPO

**Authors:** Rui Zheng, Shihan Dou, Songyang Gao, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Limao Xiong, Lu Chen, Zhiheng Xi, Yuhao Zhou, Nuo Xu, Wenbin Lai, Minghao Zhu, Rongxiang Weng, Wensen Cheng, Cheng Chang, Zhangyue Yin, Yuan Hua  
**arXiv:** [2307.04964](https://arxiv.org/abs/2307.04964)  
**Date:** 2023-07-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.04964v1/

## Key points


## Summary

Large language models (LLMs) have become a crucial component in the development of artificial general intelligence, with the goal of creating human-centric assistants. Reinforcement learning with human feedback (RLHF) is a key technological paradigm for achieving alignment with humans. However, there are several limitations and challenges that need to be addressed. One limitation is the scaling law, as the impact of model size and data scale on RLHF performance has not been thoroughly investigated. The current study focuses on a 7-billion-parameter model, but further research is needed to understand how different sizes and scales affect RLHF. Another limitation is the reward model used in RLHF experiments. The study relies on openly available English human preference datasets and a small amount of self-constructed Chinese data. While these datasets provide some evaluation of the reward model, they may not be sufficient for a comprehensive assessment. The evaluation metric used in this study primarily relies on manual evaluations and automated evaluations using GPT-4. While these methods provide some insights into RLHF abilities, there are numerous benchmarks and NLP tasks that could be utilized for a more detailed assessment. During the Proximal Policy Optimization (PPO) phase, the focus is more on achieving stability rather than enhancing final performance. While stability is important, it does not guarantee improved outcomes; additionally, the reward score alone may not reliably predict RLHF performance during training, indicating a need for a more suitable performance indicator. Despite these limitations, there have been significant contributions made in this study. Competitive Chinese and English reward models have been released with good cross-model generalization ability, reducing the cost of relabeling human preference data. In-depth analysis of the PPO algorithm has led to the proposal of an advanced version called PPO-max which ensures stable model training; moreover complete PPO-max codes have also been released to facilitate better alignment between LLMs and humans. Overall while there are still challenges and limitations in exploring RLHF this study provides valuable insights and contributions towards the development of human-aligned LLMs.