# Dueling Posterior Sampling for Preference-Based Reinforcement Learning

**Authors:** Ellen R. Novoseller, Yanan Sui, Yisong Yue, Joel W. Burdick  
**arXiv:** [1908.01289](https://arxiv.org/abs/1908.01289)  
**Date:** 2019-08-04  
**Summary page:** https://www.summarizepaper.com/en/paper/1908.01289v1/

## Key points

- Preference-based reinforcement learning (RL) focuses on agents receiving preferences rather than absolute feedback
- Dueling Posterior Sampling (DPS) is introduced by a team of researchers including Ellen R. Novoseller, Yanan Sui, Yisong Yue, and Joel W. Burdick
- DPS combines preference-based bandit learning and posterior sampling in RL to learn system dynamics and utility function
- DPS handles trajectory-level preferences using preference-based posterior sampling
- Bayesian framework is used for credit assignment translating user preferences into a posterior distribution over state/action reward models
- An asymptotic no-regret rate is established for DPS using a Bayesian logistic regression credit assignment model
- Empirical evaluations show competitive performance of DPS against existing baselines in practical scenarios

## Summary

In the realm of preference-based reinforcement learning (RL), where agents receive preferences rather than absolute feedback while interacting with the environment, there is a growing interest in developing formal frameworks that allow for tractable theoretical analysis. Addressing this challenge, a team of researchers including Ellen R. Novoseller, Yanan Sui, Yisong Yue, and Joel W. Burdick from institutions such as California Institute of Technology and Stanford University introduce Dueling Posterior Sampling (DPS). This innovative approach builds upon concepts from preference-based bandit learning and posterior sampling in RL to enable the agent to learn both the system dynamics and the underlying utility function governing user preferences. Unlike traditional RL settings where feedback is provided on individual state/action pairs, DPS leverages preference-based posterior sampling to handle trajectory-level preferences. By employing a Bayesian framework to tackle the credit assignment problem, user preferences are translated into a posterior distribution over state/action reward models. The researchers establish an asymptotic no-regret rate for DPS using a Bayesian logistic regression credit assignment model, marking a significant advancement as it represents the first regret guarantee for preference-based RL. Furthermore, the team explores potential avenues for extending their proof methodology to analyze alternative credit assignment models in future research endeavors. Empirical evaluations demonstrate that DPS exhibits competitive performance when compared against existing baselines in practical scenarios. With 35 pages in total including figures and tables, this work sheds light on cutting-edge developments in preference-based RL and sets a foundation for further advancements in this field.