# BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues

**Authors:** Haodong Duan, Jueqi Wei, Chonghua Wang, Hongwei Liu, Yixiao Fang, Songyang Zhang, Dahua Lin, Kai Chen  
**arXiv:** [2310.13650](https://arxiv.org/abs/2310.13650)  
**Date:** 2023-10-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.13650v1/

## Key points

- Study title: "BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues"
- Authors: Haodong Duan, Jueqi Wei, Chonghua Wang, Hongwei Liu, Yixiao Fang, Songyang Zhang, Dahua Lin, and Kai Chen
- Challenge of manually evaluating large language models (LLMs) in multi-turn dialogues
- Proposal of an innovative LLM-based approach for assessing performance in human-style chatting scenarios
- Methodology involving using real-world human dialogues to create ChatSEED for LLMs to generate complete multi-turn dialogues
- Use of GPT-4 as a judge to evaluate dialogue quality
- GPT-4's proficiency in producing human-style multi-turn dialogues surpassing other models
- Difficulty for discriminators to differentiate between GPT-4 generated dialogues and authentic human interactions
- Challenges faced by other LLMs in generating satisfactory multi-turn dialogues due to issues like poor instruction-following abilities or lengthy utterances
- Availability of data and codes on GitHub for further exploration and evaluation of conversational capabilities of large language models

## Summary

In the study titled "BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues," authors Haodong Duan, Jueqi Wei, Chonghua Wang, Hongwei Liu, Yixiao Fang, Songyang Zhang, Dahua Lin, and Kai Chen delve into the realm of large language models (LLMs) and their ability to engage in high-quality multi-turn dialogues with humans. The authors highlight the challenge of manually evaluating such capabilities and propose an innovative LLM-based approach for assessing the performance of existing models in human-style chatting scenarios. The methodology involves using real-world human dialogues as a starting point and extracting the initial utterances as ChatSEED. Subsequently, LLMs are tasked with generating complete multi-turn dialogues based on the ChatSEED one utterance at a time. State-of-the-art LLMs like GPT-4 are then employed as judges to evaluate the quality of the generated dialogues through various evaluation protocols. The results reveal that GPT-4 demonstrates remarkable proficiency in producing human-style multi-turn dialogues, surpassing its counterparts significantly. Discriminators struggle to differentiate between GPT-4 generated dialogues and authentic human interactions. On the contrary, other LLMs face challenges in generating satisfactory multi-turn dialogues due to issues such as poor instruction-following abilities or tendency to produce lengthy utterances. Overall, this study provides valuable insights into the capabilities of LLMs in engaging in multi-turn conversations resembling human interactions. The data and codes associated with this research are made available on GitHub for further exploration and serve as a valuable resource for evaluating the conversational prowess of large language models.