# Deep reinforcement learning from human preferences

**Authors:** Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei  
**arXiv:** [1706.03741](https://arxiv.org/abs/1706.03741)  
**Date:** 2017-06-12  
**Summary page:** https://www.summarizepaper.com/en/paper/1706.03741v1/

## Key points

- Reinforcement learning (RL) has made significant progress, but communicating complex goals to RL systems remains a challenge.
- Researchers led by Paul Christiano and Jan Leike have developed an innovative approach to this problem in their study titled "Deep Reinforcement Learning from Human Preferences."
- The method involves defining goals based on non-expert human preferences between pairs of trajectory segments.
- By providing feedback on less than one percent of the agent's interactions with the environment, they were able to reduce the cost of human oversight enough that it could be practically applied to state-of-the-art RL systems.
- This approach was effective in solving complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion.
- The approach is flexible and can handle considerably more complex behaviors and environments than any previously learned from human feedback.
- The authors note that their work has important implications for developing sophisticated RL systems that can interact usefully with real-world environments.
- This approach may help pave the way for advances in fields such as robotics, autonomous vehicles, and more.

## Summary

The field of reinforcement learning (RL) has made significant strides in recent years, but one major challenge remains: how to communicate complex goals to RL systems so that they can interact effectively with real-world environments. In a new study titled "Deep Reinforcement Learning from Human Preferences," a team of researchers led by Paul Christiano and Jan Leike explore an innovative approach to this problem. Their method involves defining goals in terms of non-expert human preferences between pairs of trajectory segments. By providing feedback on less than one percent of the agent's interactions with the environment, they were able to reduce the cost of human oversight enough that it could be practically applied to state-of-the-art RL systems. This approach was shown to be effective in solving complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion. One key advantage of this approach is its flexibility. The researchers were able to successfully train complex novel behaviors with about an hour of human time, demonstrating that their method can handle considerably more complex behaviors and environments than any previously learned from human feedback. The authors note that their work has important implications for developing sophisticated RL systems that can interact usefully with real-world environments. By enabling humans to communicate complex goals more effectively, this approach may help pave the way for advances in fields such as robotics, autonomous vehicles, and more.