AI summary · arXiv: 2309.11998v4 · cs.CL

LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li,
and 5 more authorsZi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, Hao Zhang
Original paper Submitted to arXiv on 21 Sep. 2023 License: NONEXCLUSIVE-DISTRIB 1.0

How to read this summary? Illustrated guide

The essentials

  • Authors emphasize the significance of studying human interactions with large language models (LLMs) in real-world settings
  • Introduction of the LMSYS-Chat-1M dataset containing one million authentic conversations involving 25 cutting-edge LLMs
  • Dataset gathered from 210K distinct IP addresses through Vicuna demo and Chatbot Arena website
  • Noteworthy aspects highlighted include diversity, scale, and its value for developing content moderation models and training instruction-following models
  • Emphasis on the dataset as a pivotal tool for advancing LLM capabilities by offering insights into user interactions
  • Public availability of the LMSYS-Chat-1M dataset at https://huggingface.co/datasets/lmsys/lmsys-chat-1m

Key points written by AI: they may contain errors, check them against the paper.

AI-generated keywords: LLM conversation dataset large language models real-world settings human interactions LMSYS-Chat-1M

Read further

  • Comprehensive Summary
  • Layman's Summary
  • Blog article
  • Original abstract
In their paper titled "LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset," authors Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica and Hao Zhang delve into the significance of studying human interactions with large language models (LLMs) in real-world settings. With the increasing prevalence of LLMs across various applications , understanding how individuals engage with these models is crucial. The authors introduce the as a substantial resource containing one million authentic conversations involving 25 cutting-edge LLMs. This extensive dataset was gathered from 210K distinct IP addresses through the Vicuna demo and Chatbot Arena website. The paper provides an in-depth overview of the dataset's contents , as well as its topic distribution. Noteworthy aspects such as diversity , and scale are highlighted to underscore the dataset's value. Furthermore These include developing content moderation models that rival GPT-4 performance levels training instruction-following models akin to Vicuna capabilities It is emphasized that this dataset serves as a pivotal tool for advancing LLM capabilities by offering insights into user interactions. The paper concludes by emphasizing the public availability of the LMSYS-Chat-1M dataset at https://huggingface.co/datasets/lmsys/lmsys-chat-1m. Overall, this comprehensive study sheds light on the importance of real-world interaction data for enhancing our understanding and progress in leveraging large language models effectively across diverse applications.

Was this synthesis useful?

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Figures and plots

Ask about a figure by number or use its question button to request visual analysis. Up to two figures and four image panels per question. Values read from plots are approximate; check the original data.

AI assistant instructions?

Tools for this paper

Explain it differently 1 free rewrite left today
Save to a collection Group papers in collections to compare them, chat with them and export them. Continue with Google Create a free account Log in

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero Zotero guide
Citation alerts More than 300 papers already cite it: be emailed each new one. Citation alerts guide
Weekly letter The latest letter of its field, "Artificial intelligence this week" of 16 November 2026: Artificial intelligence this week: What impressive scores leave out · Receive one by email for free

Papers around this one

See them on a map

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

  1. Very close Same question Cites this paper
    ShareChat: A Dataset of Chatbot Conversations in the Wild
    Yueru Yan, Tuc Nguyen, Bo Su et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 17 times · Not summarized yet

    ShareChat collects real-world chatbot conversations across platforms, like LMSYS-Chat-1M, but preserves platform-specific features and covers more languages.

    Abstract

    By evaluating Large Language Models (LLMs) through uniform, text-only interfaces, current academic benchmarks obscure how the unique designs and affordances of distinct commercial platforms shape real-world user behavior and system performance. To bridge this gap, we present ShareChat, the first large-scale corpus of 142,808 conversations (660,293 turns) collected from publicly shared URLs on ChatGPT, Perplexity, Grok, Gemini, and Claude. ShareChat preserves native platform affordances, including citations, thinking traces, and code artifacts, across 95 languages and the period from April 2023 to October 2025, complementing existing corpora that homogenize these interactions. To demonstrate …

  2. Very close Another approach to the problem
    BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues
    Haodong Duan, Jueqi Wei, Chonghua Wang et al. · 2023 · Computation and Language · preprint, not peer reviewed · cited 48 times · AI summary available

    BotChat evaluates multi-turn dialogue capability by prompting LLMs to generate conversations from real dialogue openings, rather than releasing a corpus of observed LLM chats.

    Abstract

    Interacting with human via high-quality multi-turn dialogues is a key feature of large language models (LLMs). However, human-based evaluation of such capability involves intensive manual labor. This report provides a preliminary evaluation of existing large language models for human-style multi-turn chatting, through an LLM-based approach. We start from real-world human dialogues and keep the very first utterances as the ChatSEED. Then we prompt LLMs to generate a full multi-turn dialogue (tens of utterances) based on the ChatSEED, utterance by utterance. Finally, we adopt state-of-the-art LLMs (GPT-4, \etc) as the judge to evaluate the generated dialogues. With different evaluation protoco…

  3. Very close Review Cites this paper
    Datasets for Large Language Models: A Comprehensive Survey
    Yang Liu, Jiahuan Cao, Chongyu Liu et al. · 2024 · Computation and Language · cited 119 times · AI summary available

    Datasets for Large Language Models surveys dataset categories and trends, citing LMSYS-Chat-1M as one resource in the broader landscape.

    Abstract

    This paper embarks on an exploration into the Large Language Model (LLM) datasets, which play a crucial role in the remarkable advancements of LLMs. The datasets serve as the foundational infrastructure analogous to a root system that sustains and nurtures the development of LLMs. Consequently, examination of these datasets emerges as a critical topic in research. In order to address the current lack of a comprehensive overview and thorough analysis of LLM datasets, and to gain insights into their current status and future trends, this survey consolidates and categorizes the fundamental aspects of LLM datasets from five perspectives: (1) Pre-training Corpora; (2) Instruction Fine-tuning Data…

  4. Very close Builds on this paper Cites this paper
    Decoding Human-LLM Collaboration in Coding: An Empirical Study of Multi-Turn Conversations in the Wild
    Binquan Zhang, Li Zhang, Hao-Yuan Zhang et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 4 times · Not summarized yet

    Decoding Human-LLM Collaboration analyzes coding interactions using LMSYS-Chat-1M and WildChat to study collaboration and instruction following.

    Abstract

    Large language models (LLMs) are increasingly acting as dynamic conversational interfaces, supporting multi-turn interactions that mimic human-like conversation and facilitate complex tasks like coding. While datasets such as LMSYS-Chat-1M and WildChat capture real-world user-LLM conversations, few studies systematically explore the mechanisms of human-LLM collaboration in coding scenarios. What tortuous paths do users experience during the interaction process? How well do the LLMs follow instructions? Are users satisfied? In this paper, we conduct an empirical analysis on human-LLM coding collaboration using LMSYS-Chat-1M and WildChat datasets to explore the human-LLM collaboration mechanis…

  5. Close Broader context
    Building Emotional Support Chatbots in the Era of LLMs
    Zhong-Hua Zheng, Li-Zi Liao, Yang Deng et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 97 times · Not summarized yet

    Building Emotional Support Chatbots creates emotional-support dialogue data for chatbot training, a specialized application distinct from LMSYS-Chat-1M's broad conversation corpus.

    Abstract

    The integration of emotional support into various conversational scenarios presents profound societal benefits, such as social interactions, mental health counseling, and customer service. However, there are unsolved challenges that hinder real-world applications in this field, including limited data availability and the absence of well-accepted model training paradigms. This work endeavors to navigate these challenges by harnessing the capabilities of Large Language Models (LLMs). We introduce an innovative methodology that synthesizes human insights with the computational prowess of LLMs to curate an extensive emotional support dialogue dataset. Our approach is initiated with a meticulousl…

  6. Close Review
    LLMOps in Action: A Framework for Designing, Deploying, and Governing Advanced Chatbots
    Pradheepan Raghavan, D. Tamburri, Stefano Fossati et al. · 2026 · IEEE Transactions on Knowledge and Data Engineering · Not summarized yet

    LLMOps in Action reviews chatbot architectures, practices, and evaluation across domains rather than introducing a real-world conversation dataset.

    Abstract

    This paper presents a systematic literature review (SLR) focused on the implementation of chatbots using Large Language Models (LLMs), aimed at providing insights into the architectures, frameworks, best practices, and evaluation metrics that are shaping the field. By analyzing 39 primary studies, the review addresses six key research questions, exploring common architectures such as client-server and Retrieval-Augmented Generation (RAG), and identifying frequently utilized models, including the GPT family, BERT, and open-source models like LLaMA. The paper evaluates the performance of these models across various domains, emphasizing the impact of fine-tuning, prompt engineering, and embeddi…

  7. Close Builds on this paper Cites this paper
    Linguistic anthropomorphism in conversational AI: theoretical foundations and empirical evidence from LMSYS chat 1M
    J. L. Blanco-Guzmán, P. Palos-Sánchez, Dan-Andrei Sitar-Tǎut et al. · 2026 · Online information review (Print) · Not summarized yet

    Linguistic anthropomorphism in conversational AI analyzes language markers in LMSYS-Chat-1M conversations across models and languages.

    Abstract

    This study investigates linguistic anthropomorphism in large language models (LLMs) by analyzing over one million conversational turns generated by 25 state-of-the-art models, including GPT-4, Claude, Vicuna, Llama-2, and others. The research examines anthropomorphic language markers across eight major languages and explores variation in anthropomorphic language use by model and language. Using the Large Model Systems (LMSYS) Chat 1M dataset from Hugging Face, the study examines anthropomorphic language markers such as first-person pronouns, explicit empathy statements, and emojis across eight major languages. The methodology combines the detection of predefined anthropomorphic and empatheti…

  8. Close Broader context
    llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodology
    Masanori Hirano, Masahiro Suzuki, Hiroki Sakaji · 2023 · Computation and Language · cited 9 times · Not summarized yet

    llm-japanese-dataset constructs Japanese task data for tuning LLMs, whereas LMSYS-Chat-1M collects real-world conversations across models.

    Abstract

    This study constructed a Japanese chat dataset for tuning large language models (LLMs), which consist of about 8.4 million records. Recently, LLMs have been developed and gaining popularity. However, high-performing LLMs are usually mainly for English. There are two ways to support languages other than English by those LLMs: constructing LLMs from scratch or tuning existing models. However, in both ways, datasets are necessary parts. In this study, we focused on supporting Japanese in those LLMs and making a dataset for training or tuning LLMs in Japanese. The dataset we constructed consisted of various tasks, such as translation and knowledge tasks. In our experiment, we tuned an existing L…

Prior work it builds on

  1. Close Groundwork for this paper A method it uses
    The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems
    Ryan Lowe, Nissan Pow, I. Serban et al. · 2015 · SIGDIAL Conference · cited 1,128 times · Not summarized yet

    The Ubuntu Dialogue Corpus provides a large collection of unstructured multi-turn dialogues and is cited as an earlier resource for dialogue research.

    Abstract

    This paper introduces the Ubuntu Dialogue Corpus, a dataset containing almost 1 million multi-turn dialogues, with a total of over 7 million utterances and 100 million words. This provides a unique resource for research into building dialogue managers based on neural language models that can make use of large amounts of unlabeled data. The dataset has both the multi-turn property of conversations in the Dialog State Tracking Challenge datasets, and the unstructured nature of interactions from microblog services such as Twitter. We also describe two neural learning architectures suitable for analyzing this dataset, and provide benchmark performance on the task of selecting the best next respo…

  2. Close Groundwork for this paper A method it uses
    MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling
    Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng et al. · 2018 · Conference on Empirical Methods in Natural Language Processing · cited 1,666 times · Not summarized yet

    MultiWOZ releases a large, labeled human-human task-dialogue corpus and is cited as earlier dataset work addressing limited dialogue data.

    Abstract

    Even though machine learning has become the major scene in dialogue research community, the real breakthrough has been blocked by the scale of data available.To address this fundamental obstacle, we introduce the Multi-Domain Wizard-of-Oz dataset (MultiWOZ), a fully-labeled collection of human-human written conversations spanning over multiple domains and topics.At a size of 10k dialogues, it is at least one order of magnitude larger than all previous annotated task-oriented corpora.The contribution of this work apart from the open-sourced dataset is two-fold:firstly, a detailed description of the data collection procedure along with a summary of data structure and analysis is provided. The …

  3. Close Review Cites this paper
    A Survey on Evaluation of Large Language Models
    Yupeng Chang, Xu Wang, Jindong Wang et al. · 2023 · Computation and Language · cited 3,999 times · AI summary available

    A Survey on Evaluation of Large Language Models reviews evaluation dimensions and methods and cites LMSYS-Chat-1M.

    Abstract

    Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide an overview from the …

  4. Related Groundwork for this paper Used in depth by this paper
    OpenAssistant Conversations - Democratizing Large Language Model Alignment
    Andreas Kopf, Yannic Kilcher, Dimitri von Rütte et al. · 2023 · Neural Information Processing Systems · cited 983 times · Not summarized yet

    OpenAssistant Conversations releases human-feedback conversation data for LLM alignment and is cited as an earlier dataset resource.

    Abstract

    Aligning large language models (LLMs) with human preferences has proven to drastically improve usability and has driven rapid adoption as demonstrated by ChatGPT. Alignment techniques such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) greatly reduce the required skill and domain knowledge to effectively harness the capabilities of LLMs, increasing their accessibility and utility across various domains. However, state-of-the-art alignment techniques like RLHF rely on high-quality human feedback data, which is expensive to create and often remains proprietary. In an effort to democratize research on large-scale alignment, we release OpenAssistant Convers…

  5. Close Groundwork for this paper
    Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
    Ning Ding, Yulin Chen, Bo-Kai Xu et al. · 2023 · Conference on Empirical Methods in Natural Language Processing · cited 1,075 times · Not summarized yet

    UltraChat provides synthetic instructional conversations for fine-tuning chat models and is cited as earlier conversation-data work.

    Abstract

    Fine-tuning on instruction data has been widely validated as an effective practice for implementing chat language models like ChatGPT. Scaling the diversity and quality of such data, although straightforward, stands a great chance of leading to improved performance. This paper aims to improve the upper bound of open-source models further. We first provide a systematically designed, diverse, informative, large-scale dataset of instructional conversations, UltraChat, which does not involve human queries. Our objective is to capture the breadth of interactions that a human might have with an AI assistant and employs a comprehensive framework to generate multi-turn conversation iteratively. Ultr…

Papers that cite it

  1. Close Builds on this paper Uses this paper in depth
    Investigating Thematic Patterns and User Preferences in LLM Interactions using BERTopic
    Abhay Bhandarkar, Gaurav Mishra, Khushi Juchani et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet

    Investigating Thematic Patterns and User Preferences applies BERTopic to LMSYS-Chat-1M to examine conversation topics and model preferences.

    Abstract

    This study applies BERTopic, a transformer-based topic modeling technique, to the lmsys-chat-1m dataset, a multilingual conversational corpus built from head-to-head evaluations of large language models (LLMs). Each user prompt is paired with two anonymized LLM responses and a human preference label, used to assess user evaluation of competing model outputs. The main objective is uncovering thematic patterns in these conversations and examining their relation to user preferences, particularly if certain LLMs are consistently preferred within specific topics. A robust preprocessing pipeline was designed for multilingual variation, balancing dialogue turns, and cleaning noisy or redacted data.…

  2. Close Builds on this paper Uses this paper in depth
    Examining Human-Like Behaviors in LLMs: A Multi-Dimensional Analysis of Model Behaviors, User Factors, and System Prompts
    Sunnie S. Y. Kim, M. Bowler, Leon A. Gatys · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet

    Examining Human-Like Behaviors analyzes human-like behaviors in multi-turn LLM conversations and cites LMSYS-Chat-1M.

    Abstract

    Large language models (LLMs) exhibit a wide range of human-like behaviors, from expressing thoughts and emotions, to engaging in relationship-building with users, to refusing requests and maintaining boundaries. Despite their prevalence, researchers and practitioners lack methods and empirical insights to make informed decisions about when and what types of human-like behaviors LLMs should exhibit. To fill this gap, we present a multi-dimensional analysis of the prevalence, potential effects, and controllability of these behaviors using LLM-as-a-judge and human evaluation. Across 21,000 multi-turn conversations from four widely used models (gpt-4o, gpt-4.1-mini, claude-sonnet-4.6, gemini-2.5…

  3. Close Builds on this paper Uses this paper in depth
    Dataset Signatures in Human-LLM Interactions and User Modeling
    Joseph Suh, Serina Chang · 2026 · Not summarized yet

    Dataset Signatures compares human-LLM interaction datasets, including LMSYS-Chat-1M, to study how dataset origins shape observed interactions.

    Abstract

    Human--LLM interaction datasets shape our understanding of AI use and provide a foundation for downstream research, including training and evaluation of user models. In recent years, a growing number of datasets have sought to capture a representative picture of human--LLM interactions. But how different are the pictures these datasets provide, and what do those differences mean for research built on them? We study these questions across seven conversation datasets, spanning in-the-wild chat logs and human preference data. We begin by revisiting the dataset classification experiment of Torralba&Efros and find that neural network classifiers identify the source of a conversation from user mes…

  4. Close Broader context
    AI-Wrapped: Participatory, Privacy-Preserving Measurement of Longitudinal LLM Use In-the-Wild
    Cathy Mengying Fang, Sheer Karny, Chayapatr Archiwaranguprok et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 5 times · Not summarized yet

    AI-Wrapped collects privacy-preserving longitudinal chatbot histories from participants, addressing naturalistic LLM use through a different collection approach.

    Abstract

    Alignment research on large language models (LLMs) increasingly depends on understanding how these systems are used in everyday contexts. Yet naturalistic interaction data is difficult to access due to privacy constraints and platform control. We present AI-Wrapped, a prototype workflow for collecting naturalistic LLM chatbot usage data while providing participants with an immediate"wrapped"-style report on their usage statistics, top topics, and behavioral patterns. We report findings from an initial deployment with 82 U.S.-based adults across 48,495 conversations from their 2025 chat histories. Participants used LLMs for both instrumental and reflective purposes and had topics with emotion…

  5. Close Broader context
    @GrokSet: multi-party Human-LLM Interactions in Social Media
    Matteo Migliarini, Berat Ercevik, Oluwagbemike Olowe et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet

    @GrokSet collects public multi-party interactions involving Grok on social media, unlike LMSYS-Chat-1M's chatbot-site conversations.

    Abstract

    Large Language Models (LLMs) are increasingly deployed as active participants on public social media platforms, yet their behavior in these unconstrained social environments remains largely unstudied. Existing datasets, drawn primarily from private chat interfaces, lack the multi-party dynamics and public visibility crucial for understanding real-world performance. To address this gap, we introduce @GrokSet, a large-scale dataset of over 1 million tweets involving the @Grok LLM on X. Our analysis reveals a distinct functional shift: rather than serving as a general assistant, the LLM is frequently invoked as an authoritative arbiter in high-stakes, polarizing political debates. However, we o…

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

Updated on 2 November 2026.

The illustrated guide to the papers around a paper

The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.