# RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

**Authors:** Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, Sushant Prakash  
**arXiv:** [2309.00267](https://arxiv.org/abs/2309.00267)  
**Date:** 2023-09-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.00267v3/

## Key points

- Reinforcement Learning from AI Feedback (RLAIF) is a new approach that trains the reward model on preferences generated by an off-the-shelf large language model (LLM).
- RLAIF achieved comparable performance to Reinforcement Learning from human feedback (RLHF) across tasks such as summarization, helpful dialogue generation, and harmless dialogue generation.
- The alignment of the AI labeler increased as the size of the LLM labeler increased, suggesting that scaling up the AI labeler size may lead to higher quality preference labels without significantly increasing costs.
- Qualitative observations comparing RLAIF and RLHF for the summarization task revealed differences in their outputs: RLHF sometimes hallucinated information not present in the original text, while RLAIF occasionally produced less fluent summaries.
- RLAIF shows promise as a scalable solution for training language models without relying heavily on human feedback, leveraging AI-generated preferences and potentially larger AI labelers to achieve performance on par with RLHF.
- Techniques like direct-RLAIF have shown superior performance compared to traditional RLAIF methods, highlighting the potential of RLAIF in overcoming scalability limitations associated with RLHF and advancing self-improvement in language model training processes.

## Summary

, , , , 

Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences. However, the cost of gathering high-quality preference labels remains a limiting factor. A new approach called Reinforcement Learning from AI Feedback (RLAIF) offers a promising alternative by training the reward model (RM) on preferences generated by an off-the-shelf LLM. In a study comparing RLAIF to RLHF across tasks such as summarization, helpful dialogue generation, and harmless dialogue generation, it was found that RLAIF achieved comparable performance to RLHF. One interesting finding was that the alignment of the AI labeler increased as the size of the LLM labeler increased. This suggests that scaling up the AI labeler size may lead to even higher quality preference labels without significantly increasing costs. In qualitative observations comparing RLAIF and RLHF for the summarization task, it was noted that while both policies produced similar summaries in many cases, there were instances where they diverged. RLHF sometimes hallucinated information not present in the original text, while RLAIF occasionally produced less fluent summaries. Further evaluation is needed to determine if these patterns exist at scale. Overall, RLAIF shows promise as a scalable solution for training language models without relying heavily on human feedback. By leveraging AI-generated preferences and potentially larger AI labelers, RLAIF can achieve performance on par with RLHF. Additionally, techniques like direct-RLAIF have shown superior performance compared to traditional RLAIF methods. These findings highlight the potential of RLAIF in overcoming scalability limitations associated with RLHF and advancing self-improvement in language model training processes.