# Statistical Rejection Sampling Improves Preference Optimization

**Authors:** Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, Jialu Liu  
**arXiv:** [2309.06657](https://arxiv.org/abs/2309.06657)  
**Date:** 2023-09-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.06657v2/

## Key points

- Alignment of language models with human preferences is a significant research challenge
- Previous approaches mainly used Reinforcement Learning from Human Feedback (RLHF) with methods like Proximal Policy Optimization (PPO)
- Offline methods such as Sequence Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have emerged as promising alternatives
- SLiC refines its loss function using sequence pairs from a supervised fine-tuned policy, while DPO directly optimizes language models based on preference data
- Both SLiC and DPO face limitations in accurately estimating the optimal policy
- A novel approach called Statistical Rejection Sampling Optimization (RSO) aims to extract preference data from the target optimal policy using rejection sampling for improved estimation
- RSO consistently outperforms both SLiC and DPO in evaluations conducted with Large Language Models (LLMs) and human raters
- A unified framework has been proposed to enhance loss functions used in both SLiC and DPO from a preference modeling perspective

## Summary

, , , , 

In recent years, the alignment of language models with human preferences has become a significant research challenge. Previous approaches have mainly relied on Reinforcement Learning from Human Feedback (RLHF) using online RL methods like Proximal Policy Optimization (PPO). However, more recently, offline methods such as Sequence Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have emerged as promising alternatives. These methods offer enhanced stability and scalability while maintaining competitive performance. SLiC refines its loss function by utilizing sequence pairs sampled from a supervised fine-tuned (SFT) policy. On the other hand, DPO directly optimizes language models based on preference data without requiring a separate reward model. Despite their advantages, both SLiC and DPO face limitations in accurately estimating the optimal policy. DPO lacks a reward model to effectively sample preference pairs, while SLiC is constrained to sampling preference pairs only from the SFT policy. To address these challenges, a novel approach called Statistical Rejection Sampling Optimization (RSO) has been introduced. RSO aims to extract preference data from the target optimal policy using rejection sampling, thereby improving the estimation of the optimal policy. Additionally, a unified framework has been proposed to enhance the loss functions used in both SLiC and DPO from a preference modeling perspective. Extensive experiments across three diverse tasks have demonstrated that RSO consistently outperforms both SLiC and DPO in evaluations conducted with Large Language Models (LLMs) and human raters. This research was accepted at ICLR 2024 and was authored by Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. Overall, this study highlights the importance of refining language models to better align with human preferences through innovative optimization techniques like RSO.