# Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

**Authors:** Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan  
**arXiv:** [2307.15217](https://arxiv.org/abs/2307.15217)  
**Date:** 2023-07-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.15217v2/

## Key points

- Growing recognition of the need to address risks and limitations associated with reinforcement learning from human feedback (RLHF)
- Lack of systematic exploration of flaws in RLHF in the public domain
- Comprehensive overview of open problems and fundamental constraints inherent in RLHF and related methods
- Emphasis on transparency and accountability in AI systems developed using RLHF
- Key details that should be disclosed to mitigate risks, including descriptions of pretraining process, selection and training of human evaluators, methods for selecting feedback examples, types of human feedback used, quality assurance measures in feedback collection, loss functions for fitting reward models, evaluation results for reward models and policies, internal and external auditing processes, monitoring and handling failures post-deployment, plans for correcting emerging failures
- Advocacy for red teaming exercises to evaluate policies and identify potential risks associated with misaligned objectives or capabilities that could deceive humans
- Importance of ongoing research in AI governance to establish standardized documentation practices for documenting risks associated with AI systems developed using RLHF

## Summary

There is a growing recognition of the need to address potential risks and limitations associated with reinforcement learning from human feedback (RLHF). While RLHF has become a popular method for fine-tuning large language models (LLMs) to align with human goals, there is a lack of systematic exploration of its flaws in the public domain. This paper aims to provide a comprehensive overview of open problems and fundamental constraints inherent in RLHF and related methods. It delves into various aspects of RLHF, including obtaining human feedback and developing reward models. Emphasizing the importance of transparency and accountability in AI systems developed using RLHF, this paper highlights key details that should be disclosed to mitigate risks. These details include descriptions of the pretraining process, selection and training of human evaluators, methods for selecting feedback examples, types of human feedback used, quality assurance measures in feedback collection, loss functions for fitting reward models, evaluation results for reward models and policies, internal and external auditing processes, monitoring and handling failures post-deployment, as well as plans for correcting emerging failures. Furthermore, this paper advocates for red teaming exercises to evaluate policies and identify potential risks associated with misaligned objectives or capabilities that could deceive humans. It also underscores the importance of ongoing research in AI governance to establish standardized documentation practices for documenting risks associated with AI systems developed using RLHF. In conclusion, by addressing the limitations and risks inherent in RLHF through enhanced transparency,,  as well as proactive monitoring and correction strategies post-deployment.