Statistical Rejection Sampling Improves Preference Optimization

AI-generated keywords: Language Models

AI-generated Key Points

  • Alignment of language models with human preferences is a significant research challenge
  • Previous approaches mainly used Reinforcement Learning from Human Feedback (RLHF) with methods like Proximal Policy Optimization (PPO)
  • Offline methods such as Sequence Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have emerged as promising alternatives
  • SLiC refines its loss function using sequence pairs from a supervised fine-tuned policy, while DPO directly optimizes language models based on preference data
  • Both SLiC and DPO face limitations in accurately estimating the optimal policy
  • A novel approach called Statistical Rejection Sampling Optimization (RSO) aims to extract preference data from the target optimal policy using rejection sampling for improved estimation
  • RSO consistently outperforms both SLiC and DPO in evaluations conducted with Large Language Models (LLMs) and human raters
  • A unified framework has been proposed to enhance loss functions used in both SLiC and DPO from a preference modeling perspective
Also access our AI generated: Comprehensive summary, Lay summary, Blog-like article; or ask questions about this paper to our AI assistant.

Authors: Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, Jialu Liu

Accepted in ICLR 2024
License: CC BY 4.0

Abstract: Improving the alignment of language models with human preferences remains an active research challenge. Previous approaches have primarily utilized Reinforcement Learning from Human Feedback (RLHF) via online RL methods such as Proximal Policy Optimization (PPO). Recently, offline methods such as Sequence Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have emerged as attractive alternatives, offering improvements in stability and scalability while maintaining competitive performance. SLiC refines its loss function using sequence pairs sampled from a supervised fine-tuned (SFT) policy, while DPO directly optimizes language models based on preference data, foregoing the need for a separate reward model. However, the maximum likelihood estimator (MLE) of the target optimal policy requires labeled preference pairs sampled from that policy. DPO's lack of a reward model constrains its ability to sample preference pairs from the optimal policy, and SLiC is restricted to sampling preference pairs only from the SFT policy. To address these limitations, we introduce a novel approach called Statistical Rejection Sampling Optimization (RSO) that aims to source preference data from the target optimal policy using rejection sampling, enabling a more accurate estimation of the optimal policy. We also propose a unified framework that enhances the loss functions used in both SLiC and DPO from a preference modeling standpoint. Through extensive experiments across three diverse tasks, we demonstrate that RSO consistently outperforms both SLiC and DPO on evaluations from both Large Language Model (LLM) and human raters.

Submitted to arXiv on 13 Sep. 2023

Ask questions about this paper to our AI assistant

You can also chat with multiple papers at once here.

AI assistant instructions?

Results of the summarizing process for the arXiv paper: 2309.06657v2

, , , , In recent years, the alignment of language models with human preferences has become a significant research challenge. Previous approaches have mainly relied on Reinforcement Learning from Human Feedback (RLHF) using online RL methods like Proximal Policy Optimization (PPO). However, more recently, offline methods such as Sequence Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have emerged as promising alternatives. These methods offer enhanced stability and scalability while maintaining competitive performance. SLiC refines its loss function by utilizing sequence pairs sampled from a supervised fine-tuned (SFT) policy. On the other hand, DPO directly optimizes language models based on preference data without requiring a separate reward model. Despite their advantages, both SLiC and DPO face limitations in accurately estimating the optimal policy. DPO lacks a reward model to effectively sample preference pairs, while SLiC is constrained to sampling preference pairs only from the SFT policy. To address these challenges, a novel approach called Statistical Rejection Sampling Optimization (RSO) has been introduced. RSO aims to extract preference data from the target optimal policy using rejection sampling, thereby improving the estimation of the optimal policy. Additionally, a unified framework has been proposed to enhance the loss functions used in both SLiC and DPO from a preference modeling perspective. Extensive experiments across three diverse tasks have demonstrated that RSO consistently outperforms both SLiC and DPO in evaluations conducted with Large Language Models (LLMs) and human raters. This research was accepted at ICLR 2024 and was authored by Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. Overall, this study highlights the importance of refining language models to better align with human preferences through innovative optimization techniques like RSO.
Created on 24 Jun. 2024

Assess the quality of the AI-generated content by voting

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Similar papers summarized with our AI tools

Navigate through even more similar papers through a

tree representation

Look for similar papers (in beta version)

By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.