# ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback

**Authors:** Zhenyu Hou, Yiin Niu, Zhengxiao Du, Xiaohan Zhang, Xiao Liu, Aohan Zeng, Qinkai Zheng, Minlie Huang, Hongning Wang, Jie Tang, Yuxiao Dong  
**arXiv:** [2404.00934](https://arxiv.org/abs/2404.00934)  
**Date:** 2024-04-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.00934v1/

## Key points

- ChatGLM-RLHF is a reinforcement learning from human feedback system designed to enhance ChatGLM's alignment with human preferences
- The system consists of three main components: collecting human preference data, training the reward model, and optimizing policies
- Challenges addressed include mitigating reward variance for large-scale training stability, implementing model parallelism with fused gradient-descent, and designing regularization constraints to prevent catastrophic forgetting in large language models (LLMs)
- Experimental results show that ChatGLM-RLHF outperforms the supervised fine-tuned (SFT) version of ChatGLM in alignment tasks, achieving an average of 15% more wins in Chinese alignment tasks
- Human evaluations support the effectiveness of RLHF, with the PPO model demonstrating advantages over the SFT model in diverse tasks like creative writing, logical reasoning, semantic analysis, language comprehension, and mathematics
- Ablation study reveals that both DPO and PPO models significantly increase response length compared to the SFT model, leading to improved automatic evaluation scores for creative writing tasks
- Insights from evaluating the reward model suggest it can guide RLHF algorithms with approximately 65% accuracy in mirroring human judgment

## Summary

, , , , 

In this paper, the authors introduce ChatGLM-RLHF, a reinforcement learning from human feedback system designed to enhance ChatGLM's alignment with human preferences. The system consists of three main components: collecting human preference data, training the reward model, and optimizing policies. Throughout the integration of ChatGLM-RLHF into production, the authors address various challenges such as mitigating reward variance for large-scale training stability, implementing model parallelism with fused gradient-descent, and designing regularization constraints to prevent catastrophic forgetting in large language models (LLMs). Experimental results show that ChatGLM-RLHF outperforms the supervised fine-tuned (SFT) version of ChatGLM in alignment tasks. For example, in Chinese alignment tasks, ChatGLM-RLHF achieves an average of 15% more wins against ChatGLM-SFT. Human evaluations further support the effectiveness of RLHF, with the PPO model demonstrating a clear advantage over the SFT model in diverse tasks like creative writing, logical reasoning, semantic analysis, language comprehension, and mathematics. Additionally, an ablation study reveals that both DPO and PPO models significantly increase response length compared to the SFT model. This increase in response length correlates with improved automatic evaluation scores for creative writing tasks. Furthermore, insights from evaluating the reward model suggest that it can guide RLHF algorithms with approximately 65% accuracy in mirroring human judgment. Overall,<keyword1>ChatGLM-RLHF</keyword1> is a successful implementation of <keyword2>reinforcement learning</keyword2> techniques to improve AI services powered by large language models. Through its three main components, the system effectively collects human preference data, trains the reward model, and optimizes policies to align with human preferences. The authors also address challenges such as mitigating reward variance and implementing model parallelism to ensure stable training for large-scale models. Experimental results demonstrate that <keyword1>ChatGLM-RLHF</keyword1> outperforms traditional supervised fine-tuning methods in <keyword3>alignment tasks</keyword3>, achieving an average of 15% more wins in Chinese alignment tasks. Human evaluations further support the effectiveness of <keyword1>ChatGLM-RLHF</keyword1>, with the PPO model showing a clear advantage over the SFT model in diverse tasks such as creative writing, logical reasoning, semantic analysis, language comprehension, and mathematics. An ablation study also reveals that both DPO and PPO models significantly increase response length compared to the SFT model. This increase in response length is correlated with improved automatic evaluation scores for creative writing tasks. Insights from evaluating the reward model suggest that it can guide RLHF algorithms with approximately 65% accuracy in mirroring human judgment. This highlights the importance of incorporating human evaluations alongside automatic evaluations to accurately assess model performance across various tasks. In conclusion,<keyword1>ChatGLM-RLHF</keyword1> provides valuable insights into aligning large language models (LLMs) with human preferences through reinforcement learning techniques. Its success in improving response length and task-specific performance underscores its potential to enhance AI services powered by LLMs. The findings emphasize the importance of incorporating human feedback into AI systems to better align them with human preferences and improve their overall performance.