# ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback

**Authors:** Zhenyu Hou, Yilin Niu, Zhengxiao Du, Xiaohan Zhang, Xiao Liu, Aohan Zeng, Qinkai Zheng, Minlie Huang, Hongning Wang, Jie Tang, Yuxiao Dong  
**arXiv:** [2404.00934](https://arxiv.org/abs/2404.00934)  
**Date:** 2024-04-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.00934v2/

## Key points

- Introduction of ChatGLM-RLHF, a reinforcement learning from human feedback (RLHF) system
- Components of the ChatGLM-RLHF pipeline: collecting human preference data, training the reward model, and optimizing policies
- Strategies used to mitigate challenges during integration into production: mitigating reward variance and implementing model parallelism
- Performance comparison showing that ChatGLM-RLHF outperforms the supervised fine-tuned version in alignment tasks
- Human evaluations favoring the PPO model within ChatGLM-32B over the SFT model
- Task-specific performance improvements observed in creative writing and programming tasks
- Advancements noted in practical programming instructions despite challenges in identifying errors within code snippets
- High tie rate recorded during human evaluations aligning with expectations
- Insights into aligning large language models with human preferences through RLHF implementations and strategies to overcome challenges

## Summary

The authors introduce ChatGLM-RLHF, a reinforcement learning from human feedback (RLHF) system designed to improve the alignment of ChatGLM with human preferences. The ChatGLM-RLHF pipeline consists of three main components: collecting human preference data, training the reward model, and optimizing policies. Challenges encountered during integration into production were mitigated through strategies such as mitigating reward variance for stable large-scale training and implementing model parallelism. Experiments show that ChatGLM-RLHF outperforms the supervised fine-tuned version in alignment tasks. Human evaluations were incorporated to assess effectiveness, with results showing a distinct advantage for the PPO model within ChatGLM-32B over the SFT model. Task-specific performance improvements were observed in creative writing and programming tasks. Despite challenges in accurately identifying errors within code snippets for programming tasks, significant advancements were noted in practical programming instructions like building an Anaconda in Linux. The high tie rate recorded during human evaluations aligns with expectations. This work provides insights into aligning large language models with human preferences through RLHF implementations and offers strategies to overcome challenges encountered during the process. Overall, it demonstrates the effectiveness of incorporating human feedback into reinforcement learning systems to enhance AI services like ChatGLM.