AI summary · arXiv: 2309.11998v4 · cs.CL
LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
The essentials
- Authors emphasize the significance of studying human interactions with large language models (LLMs) in real-world settings
- Introduction of the LMSYS-Chat-1M dataset containing one million authentic conversations involving 25 cutting-edge LLMs
- Dataset gathered from 210K distinct IP addresses through Vicuna demo and Chatbot Arena website
- Noteworthy aspects highlighted include diversity, scale, and its value for developing content moderation models and training instruction-following models
- Emphasis on the dataset as a pivotal tool for advancing LLM capabilities by offering insights into user interactions
- Public availability of the LMSYS-Chat-1M dataset at https://huggingface.co/datasets/lmsys/lmsys-chat-1m
Key points written by AI: they may contain errors, check them against the paper.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
ShareChat: A Dataset of Chatbot Conversations in the WildYueru Yan, Tuc Nguyen, Bo Su et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 17 times · Not summarized yet
ShareChat collects real-world chatbot conversations across platforms, like LMSYS-Chat-1M, but preserves platform-specific features and covers more languages.
Abstract
By evaluating Large Language Models (LLMs) through uniform, text-only interfaces, current academic benchmarks obscure how the unique designs and affordances of distinct commercial platforms shape real-world user behavior and system performance. To bridge this gap, we present ShareChat, the first large-scale corpus of 142,808 conversations (660,293 turns) collected from publicly shared URLs on ChatGPT, Perplexity, Grok, Gemini, and Claude. ShareChat preserves native platform affordances, including citations, thinking traces, and code artifacts, across 95 languages and the period from April 2023 to October 2025, complementing existing corpora that homogenize these interactions. To demonstrate …
-
BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn DialoguesHaodong Duan, Jueqi Wei, Chonghua Wang et al. · 2023 · Computation and Language · preprint, not peer reviewed · cited 48 times · AI summary available
BotChat evaluates multi-turn dialogue capability by prompting LLMs to generate conversations from real dialogue openings, rather than releasing a corpus of observed LLM chats.
Abstract
Interacting with human via high-quality multi-turn dialogues is a key feature of large language models (LLMs). However, human-based evaluation of such capability involves intensive manual labor. This report provides a preliminary evaluation of existing large language models for human-style multi-turn chatting, through an LLM-based approach. We start from real-world human dialogues and keep the very first utterances as the ChatSEED. Then we prompt LLMs to generate a full multi-turn dialogue (tens of utterances) based on the ChatSEED, utterance by utterance. Finally, we adopt state-of-the-art LLMs (GPT-4, \etc) as the judge to evaluate the generated dialogues. With different evaluation protoco…
-
Datasets for Large Language Models: A Comprehensive SurveyYang Liu, Jiahuan Cao, Chongyu Liu et al. · 2024 · Computation and Language · cited 119 times · AI summary available
Datasets for Large Language Models surveys dataset categories and trends, citing LMSYS-Chat-1M as one resource in the broader landscape.
Abstract
This paper embarks on an exploration into the Large Language Model (LLM) datasets, which play a crucial role in the remarkable advancements of LLMs. The datasets serve as the foundational infrastructure analogous to a root system that sustains and nurtures the development of LLMs. Consequently, examination of these datasets emerges as a critical topic in research. In order to address the current lack of a comprehensive overview and thorough analysis of LLM datasets, and to gain insights into their current status and future trends, this survey consolidates and categorizes the fundamental aspects of LLM datasets from five perspectives: (1) Pre-training Corpora; (2) Instruction Fine-tuning Data…
-
Decoding Human-LLM Collaboration in Coding: An Empirical Study of Multi-Turn Conversations in the WildBinquan Zhang, Li Zhang, Hao-Yuan Zhang et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 4 times · Not summarized yet
Decoding Human-LLM Collaboration analyzes coding interactions using LMSYS-Chat-1M and WildChat to study collaboration and instruction following.
Abstract
Large language models (LLMs) are increasingly acting as dynamic conversational interfaces, supporting multi-turn interactions that mimic human-like conversation and facilitate complex tasks like coding. While datasets such as LMSYS-Chat-1M and WildChat capture real-world user-LLM conversations, few studies systematically explore the mechanisms of human-LLM collaboration in coding scenarios. What tortuous paths do users experience during the interaction process? How well do the LLMs follow instructions? Are users satisfied? In this paper, we conduct an empirical analysis on human-LLM coding collaboration using LMSYS-Chat-1M and WildChat datasets to explore the human-LLM collaboration mechanis…
-
Building Emotional Support Chatbots in the Era of LLMsZhong-Hua Zheng, Li-Zi Liao, Yang Deng et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 97 times · Not summarized yet
Building Emotional Support Chatbots creates emotional-support dialogue data for chatbot training, a specialized application distinct from LMSYS-Chat-1M's broad conversation corpus.
Abstract
The integration of emotional support into various conversational scenarios presents profound societal benefits, such as social interactions, mental health counseling, and customer service. However, there are unsolved challenges that hinder real-world applications in this field, including limited data availability and the absence of well-accepted model training paradigms. This work endeavors to navigate these challenges by harnessing the capabilities of Large Language Models (LLMs). We introduce an innovative methodology that synthesizes human insights with the computational prowess of LLMs to curate an extensive emotional support dialogue dataset. Our approach is initiated with a meticulousl…
-
LLMOps in Action: A Framework for Designing, Deploying, and Governing Advanced ChatbotsPradheepan Raghavan, D. Tamburri, Stefano Fossati et al. · 2026 · IEEE Transactions on Knowledge and Data Engineering · Not summarized yet
LLMOps in Action reviews chatbot architectures, practices, and evaluation across domains rather than introducing a real-world conversation dataset.
Abstract
This paper presents a systematic literature review (SLR) focused on the implementation of chatbots using Large Language Models (LLMs), aimed at providing insights into the architectures, frameworks, best practices, and evaluation metrics that are shaping the field. By analyzing 39 primary studies, the review addresses six key research questions, exploring common architectures such as client-server and Retrieval-Augmented Generation (RAG), and identifying frequently utilized models, including the GPT family, BERT, and open-source models like LLaMA. The paper evaluates the performance of these models across various domains, emphasizing the impact of fine-tuning, prompt engineering, and embeddi…
-
Linguistic anthropomorphism in conversational AI: theoretical foundations and empirical evidence from LMSYS chat 1MJ. L. Blanco-Guzmán, P. Palos-Sánchez, Dan-Andrei Sitar-Tǎut et al. · 2026 · Online information review (Print) · Not summarized yet
Linguistic anthropomorphism in conversational AI analyzes language markers in LMSYS-Chat-1M conversations across models and languages.
Abstract
This study investigates linguistic anthropomorphism in large language models (LLMs) by analyzing over one million conversational turns generated by 25 state-of-the-art models, including GPT-4, Claude, Vicuna, Llama-2, and others. The research examines anthropomorphic language markers across eight major languages and explores variation in anthropomorphic language use by model and language. Using the Large Model Systems (LMSYS) Chat 1M dataset from Hugging Face, the study examines anthropomorphic language markers such as first-person pronouns, explicit empathy statements, and emojis across eight major languages. The methodology combines the detection of predefined anthropomorphic and empatheti…
-
llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its MethodologyMasanori Hirano, Masahiro Suzuki, Hiroki Sakaji · 2023 · Computation and Language · cited 9 times · Not summarized yet
llm-japanese-dataset constructs Japanese task data for tuning LLMs, whereas LMSYS-Chat-1M collects real-world conversations across models.
Abstract
This study constructed a Japanese chat dataset for tuning large language models (LLMs), which consist of about 8.4 million records. Recently, LLMs have been developed and gaining popularity. However, high-performing LLMs are usually mainly for English. There are two ways to support languages other than English by those LLMs: constructing LLMs from scratch or tuning existing models. However, in both ways, datasets are necessary parts. In this study, we focused on supporting Japanese in those LLMs and making a dataset for training or tuning LLMs in Japanese. The dataset we constructed consisted of various tasks, such as translation and knowledge tasks. In our experiment, we tuned an existing L…
Prior work it builds on
-
The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue SystemsRyan Lowe, Nissan Pow, I. Serban et al. · 2015 · SIGDIAL Conference · cited 1,128 times · Not summarized yet
The Ubuntu Dialogue Corpus provides a large collection of unstructured multi-turn dialogues and is cited as an earlier resource for dialogue research.
Abstract
This paper introduces the Ubuntu Dialogue Corpus, a dataset containing almost 1 million multi-turn dialogues, with a total of over 7 million utterances and 100 million words. This provides a unique resource for research into building dialogue managers based on neural language models that can make use of large amounts of unlabeled data. The dataset has both the multi-turn property of conversations in the Dialog State Tracking Challenge datasets, and the unstructured nature of interactions from microblog services such as Twitter. We also describe two neural learning architectures suitable for analyzing this dataset, and provide benchmark performance on the task of selecting the best next respo…
-
MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue ModellingPaweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng et al. · 2018 · Conference on Empirical Methods in Natural Language Processing · cited 1,666 times · Not summarized yet
MultiWOZ releases a large, labeled human-human task-dialogue corpus and is cited as earlier dataset work addressing limited dialogue data.
Abstract
Even though machine learning has become the major scene in dialogue research community, the real breakthrough has been blocked by the scale of data available.To address this fundamental obstacle, we introduce the Multi-Domain Wizard-of-Oz dataset (MultiWOZ), a fully-labeled collection of human-human written conversations spanning over multiple domains and topics.At a size of 10k dialogues, it is at least one order of magnitude larger than all previous annotated task-oriented corpora.The contribution of this work apart from the open-sourced dataset is two-fold:firstly, a detailed description of the data collection procedure along with a summary of data structure and analysis is provided. The …
-
A Survey on Evaluation of Large Language ModelsYupeng Chang, Xu Wang, Jindong Wang et al. · 2023 · Computation and Language · cited 3,999 times · AI summary available
A Survey on Evaluation of Large Language Models reviews evaluation dimensions and methods and cites LMSYS-Chat-1M.
Abstract
Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide an overview from the …
-
OpenAssistant Conversations - Democratizing Large Language Model AlignmentAndreas Kopf, Yannic Kilcher, Dimitri von Rütte et al. · 2023 · Neural Information Processing Systems · cited 983 times · Not summarized yet
OpenAssistant Conversations releases human-feedback conversation data for LLM alignment and is cited as an earlier dataset resource.
Abstract
Aligning large language models (LLMs) with human preferences has proven to drastically improve usability and has driven rapid adoption as demonstrated by ChatGPT. Alignment techniques such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) greatly reduce the required skill and domain knowledge to effectively harness the capabilities of LLMs, increasing their accessibility and utility across various domains. However, state-of-the-art alignment techniques like RLHF rely on high-quality human feedback data, which is expensive to create and often remains proprietary. In an effort to democratize research on large-scale alignment, we release OpenAssistant Convers…
-
Enhancing Chat Language Models by Scaling High-quality Instructional ConversationsNing Ding, Yulin Chen, Bo-Kai Xu et al. · 2023 · Conference on Empirical Methods in Natural Language Processing · cited 1,075 times · Not summarized yet
UltraChat provides synthetic instructional conversations for fine-tuning chat models and is cited as earlier conversation-data work.
Abstract
Fine-tuning on instruction data has been widely validated as an effective practice for implementing chat language models like ChatGPT. Scaling the diversity and quality of such data, although straightforward, stands a great chance of leading to improved performance. This paper aims to improve the upper bound of open-source models further. We first provide a systematically designed, diverse, informative, large-scale dataset of instructional conversations, UltraChat, which does not involve human queries. Our objective is to capture the breadth of interactions that a human might have with an AI assistant and employs a comprehensive framework to generate multi-turn conversation iteratively. Ultr…
Papers that cite it
-
Investigating Thematic Patterns and User Preferences in LLM Interactions using BERTopicAbhay Bhandarkar, Gaurav Mishra, Khushi Juchani et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet
Investigating Thematic Patterns and User Preferences applies BERTopic to LMSYS-Chat-1M to examine conversation topics and model preferences.
Abstract
This study applies BERTopic, a transformer-based topic modeling technique, to the lmsys-chat-1m dataset, a multilingual conversational corpus built from head-to-head evaluations of large language models (LLMs). Each user prompt is paired with two anonymized LLM responses and a human preference label, used to assess user evaluation of competing model outputs. The main objective is uncovering thematic patterns in these conversations and examining their relation to user preferences, particularly if certain LLMs are consistently preferred within specific topics. A robust preprocessing pipeline was designed for multilingual variation, balancing dialogue turns, and cleaning noisy or redacted data.…
-
Examining Human-Like Behaviors in LLMs: A Multi-Dimensional Analysis of Model Behaviors, User Factors, and System PromptsSunnie S. Y. Kim, M. Bowler, Leon A. Gatys · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet
Examining Human-Like Behaviors analyzes human-like behaviors in multi-turn LLM conversations and cites LMSYS-Chat-1M.
Abstract
Large language models (LLMs) exhibit a wide range of human-like behaviors, from expressing thoughts and emotions, to engaging in relationship-building with users, to refusing requests and maintaining boundaries. Despite their prevalence, researchers and practitioners lack methods and empirical insights to make informed decisions about when and what types of human-like behaviors LLMs should exhibit. To fill this gap, we present a multi-dimensional analysis of the prevalence, potential effects, and controllability of these behaviors using LLM-as-a-judge and human evaluation. Across 21,000 multi-turn conversations from four widely used models (gpt-4o, gpt-4.1-mini, claude-sonnet-4.6, gemini-2.5…
-
Dataset Signatures in Human-LLM Interactions and User ModelingJoseph Suh, Serina Chang · 2026 · Not summarized yet
Dataset Signatures compares human-LLM interaction datasets, including LMSYS-Chat-1M, to study how dataset origins shape observed interactions.
Abstract
Human--LLM interaction datasets shape our understanding of AI use and provide a foundation for downstream research, including training and evaluation of user models. In recent years, a growing number of datasets have sought to capture a representative picture of human--LLM interactions. But how different are the pictures these datasets provide, and what do those differences mean for research built on them? We study these questions across seven conversation datasets, spanning in-the-wild chat logs and human preference data. We begin by revisiting the dataset classification experiment of Torralba&Efros and find that neural network classifiers identify the source of a conversation from user mes…
-
AI-Wrapped: Participatory, Privacy-Preserving Measurement of Longitudinal LLM Use In-the-WildCathy Mengying Fang, Sheer Karny, Chayapatr Archiwaranguprok et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 5 times · Not summarized yet
AI-Wrapped collects privacy-preserving longitudinal chatbot histories from participants, addressing naturalistic LLM use through a different collection approach.
Abstract
Alignment research on large language models (LLMs) increasingly depends on understanding how these systems are used in everyday contexts. Yet naturalistic interaction data is difficult to access due to privacy constraints and platform control. We present AI-Wrapped, a prototype workflow for collecting naturalistic LLM chatbot usage data while providing participants with an immediate"wrapped"-style report on their usage statistics, top topics, and behavioral patterns. We report findings from an initial deployment with 82 U.S.-based adults across 48,495 conversations from their 2025 chat histories. Participants used LLMs for both instrumental and reflective purposes and had topics with emotion…
-
@GrokSet: multi-party Human-LLM Interactions in Social MediaMatteo Migliarini, Berat Ercevik, Oluwagbemike Olowe et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet
@GrokSet collects public multi-party interactions involving Grok on social media, unlike LMSYS-Chat-1M's chatbot-site conversations.
Abstract
Large Language Models (LLMs) are increasingly deployed as active participants on public social media platforms, yet their behavior in these unconstrained social environments remains largely unstudied. Existing datasets, drawn primarily from private chat interfaces, lack the multi-party dynamics and public visibility crucial for understanding real-world performance. To address this gap, we introduce @GrokSet, a large-scale dataset of over 1 million tweets involving the @Grok LLM on X. Our analysis reveals a distinct functional shift: rather than serving as a general assistant, the LLM is frequently invoked as an authoritative arbiter in high-stakes, polarizing political debates. However, we o…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 2 November 2026.
The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.