# LLMs Get Lost In Multi-Turn Conversation

**Authors:** Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, Jennifer Neville  
**arXiv:** [2505.06120](https://arxiv.org/abs/2505.06120)  
**Date:** 2025-05-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2505.06120v1/

## Key points

- Study focused on evaluating Large Language Models (LLMs) in multi-turn conversations vs. single-turn interactions
- Significant drop in performance in multi-turn settings, with an average decrease of 39%
- Performance degradation attributed to minor loss in aptitude and significant increase in unreliability
- LLMs tend to make assumptions early on, leading to reliance on incorrect information
- Tasks included Data-to-Text and Summary generation testing long-context capabilities
- Different metrics used for evaluation: binary correctness and continuous scoring
- Sharding process detailed in the appendix for reproducibility and future research
- Previous-generation models not designed for multi-turn conversations, but interest growing with advancements like ChatGPT
- Efforts made to evaluate LLMs' ability in longer conversations through crowd-sourced annotations and expanded frameworks

## Summary

This study focuses on evaluating the performance of Large Language Models (LLMs) in multi-turn conversations compared to single-turn interactions. The researchers conducted large-scale simulation experiments using top open- and closed-weight LLMs across six generation tasks. The results showed a significant drop in performance in multi-turn settings, with an average decrease of 39%. Analysis of over 200,000 simulated conversations revealed that the performance degradation was attributed to a minor loss in aptitude and a significant increase in unreliability. It was found that LLMs often make assumptions early on in the conversation and prematurely attempt to generate final solutions, leading to reliance on incorrect information. The study also included tasks such as Data-to-Text and Summary generation, which tested long-context capabilities known to impact model performance. Different metrics were used for evaluation, including binary correctness for certain tasks and continuous scoring for refinement tasks. The sharding process for each task was detailed in the appendix to facilitate reproducibility and future research on LLM evaluation in diverse settings. Previous-generation language models were not designed for multi-turn conversations, leading to a focus on single-turn tasks during evaluation. However, with advancements like ChatGPT sparking interest in multi-turn assessment, efforts have been made to evaluate LLMs' ability in longer conversations through crowd-sourced annotations and expanded evaluation frameworks. This study contributes valuable insights into the challenges faced by LLMs in navigating multi-turn dialogues effectively.