# Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations

**Authors:** Joey Hong, Sergey Levine, Anca Dragan  
**arXiv:** [2311.05584](https://arxiv.org/abs/2311.05584)  
**Date:** 2023-11-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.05584v1/

## Key points

- Large language models (LLMs) are effective in natural language processing tasks
- LLMs struggle with optimizing conversational outcomes in interactive goal-directed dialogue
- Proposed method adapts LLMs with reinforcement learning (RL)
- LLMs generate valuable data by simulating suboptimal but human-like behaviors
- Synthetic rollouts of hypothetical human-human interactions are used as training data
- Offline RL is employed to train an interactive conversational agent
- Empirical results show state-of-the-art performance in goal-directed dialogue tasks
- Combining LLMs and RL enables effective interactive dialogues and desired outcomes

## Summary

Large language models (LLMs) have become highly effective in various natural language processing tasks. However, when it comes to interactive applications that require goal-directed dialogue, LLMs trained with supervised fine-tuning or "single-step" reinforcement learning (RL) struggle to optimize conversational outcomes over multiple turns of interaction. In this paper, the authors propose a novel method for adapting LLMs with RL to address this limitation. The key insight of the proposed approach is that although LLMs may not effectively solve goal-directed dialogue tasks on their own, they can generate valuable data by simulating suboptimal but human-like behaviors. To leverage this potential, the authors use LLMs to sample diverse synthetic rollouts of hypothetical in-domain human-human interactions based on a textual description of a goal-directed dialogue task. Using these simulated interactions as training data, the authors employ offline reinforcement learning to train an interactive conversational agent capable of optimizing goal-directed objectives across multiple turns. This process involves using RL algorithms to process the examples generated by the LLM and learn how to perform more optimal interactions. Empirical results demonstrate that the proposed approach achieves state-of-the-art performance in various goal-directed dialogue tasks, including teaching and preference elicitation. By combining the strengths of LLMs and RL, this method enables conversational agents to effectively engage in interactive dialogues with users and achieve desired outcomes. Overall, this research contributes to advancing the field of natural language processing by addressing the challenges associated with goal-directed dialogue using large language models and reinforcement learning techniques.