# Direct Preference Optimization: Your Language Model is Secretly a Reward Model

**Authors:** Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn  
**arXiv:** [2305.18290](https://arxiv.org/abs/2305.18290)  
**Date:** 2023-05-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.18290v1/

## Key points

- Large-scale unsupervised language models (LMs) can learn broad world knowledge and reasoning skills, but controlling their behavior is challenging due to the unsupervised nature of their training.
- Existing methods for controlling LMs involve collecting human labels and fine-tuning the model with reinforcement learning from human feedback (RLHF), which is complex and unstable.
- Direct Preference Optimization (DPO) eliminates the need for fitting a reward model, sampling from the LM during fine-tuning, or significant hyperparameter tuning by solving a classification problem on human preference data.
- Experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods, exceeding RLHF's ability to control sentiment of generations and improving response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.
- The experiments explore three different open-ended text generation tasks: controlled sentiment generation using movie reviews from IMDb dataset; summarization using Reddit TL;DR summarization dataset; and single-turn dialogue using Anthropic Helpful and Harmless dialogue dataset.
- The evaluation uses two different approaches: analyzing effectiveness in optimizing constrained reward maximization objective by its frontier of achieved reward in controlled sentiment generation setting; evaluating algorithms with their win rate against baseline policy using GPT-4 as proxy for human evaluation of summary quality and response helpfulness in summarization and single-turn dialogue settings respectively.
- The results show that DPO tends to perform as well or better than strong baselines like RLHF with PPO and returning the best of N sampled trajectories under a learned reward function, with almost no tuning of hyperparameters. Additionally, DPO is the only method that improves over chosen summaries in the Anthropic-HH one-step dialogue task, while GPT-4 judgments correlate strongly with humans.

## Summary

Large-scale unsupervised language models (LMs) have the ability to learn broad world knowledge and some reasoning skills, but controlling their behavior precisely is a challenge due to the completely unsupervised nature of their training. Existing methods for achieving steerability involve collecting human labels of the relative quality of model generations and fine-tuning the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is complex and unstable, involving fitting a reward model that reflects human preferences and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In response, researchers have developed Direct Preference Optimization (DPO), which leverages a mapping between reward functions and optimal policies to optimize constrained reward maximization problems exactly with a single stage of policy training. Essentially solving a classification problem on human preference data, DPO eliminates the need for fitting a reward model, sampling from the LM during fine-tuning, or significant hyperparameter tuning. Experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds RLHF's ability to control sentiment of generations and improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train. The experiments explore three different open-ended text generation tasks: controlled sentiment generation using movie reviews from IMDb dataset; summarization using Reddit TL;DR summarization dataset; and single-turn dialogue using Anthropic Helpful and Harmless dialogue dataset. The evaluation uses two different approaches: analyzing effectiveness in optimizing constrained reward maximization objective by its frontier of achieved reward in controlled sentiment generation setting; evaluating algorithms with their win rate against baseline policy using GPT-4 as proxy for human evaluation of summary quality and response helpfulness in summarization and single-turn dialogue settings respectively. The results show that DPO tends to perform as well or better than strong baselines like RLHF with PPO and returning the best of N sampled trajectories under a learned reward function, with almost no tuning of hyperparameters. Additionally, DPO is the only method that improves over chosen summaries in the Anthropic-HH one-step dialogue task, while GPT-4 judgments correlate strongly with humans.