# How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection

**Authors:** Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, Yupeng Wu  
**arXiv:** [2301.07597](https://arxiv.org/abs/2301.07597)  
**Date:** 2023-01-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2301.07597v1/

## Key points

- ChatGPT has generated significant interest in academic and industrial communities due to its ability to provide comprehensive and fluent responses to a wide range of human questions.
- ChatGPT surpasses previous public chatbots in terms of security and usefulness.
- Concerns have been raised about the potential negative impacts that large language models (LLMs) like ChatGPT could have on society, including fake news, plagiarism, and social security issues.
- A team of researchers led by Biyang Guo conducted an extensive study comparing ChatGPT's responses with those of human experts across various domains such as open-domain, financial, medical, legal, and psychological areas.
- The Human ChatGPT Comparison Corpus (HC3) dataset was created for analysis from tens of thousands of comparison responses collected from both sources.
- The study revealed interesting insights into the characteristics of ChatGPT's responses compared to those generated by humans.
- The researchers developed three different detection systems to effectively distinguish between text generated by ChatGPT or humans.
- The HC3 dataset is publicly available along with code and models at https://github.com/Hello-SimpleAI/chatgpt-comparison-detection for further research in this area.

## Summary

The development of ChatGPT has generated significant interest in both academic and industrial communities due to its ability to provide comprehensive and fluent responses to a wide range of human questions, surpassing previous public chatbots in terms of security and usefulness. However, concerns have been raised about the potential negative impacts that large language models (LLMs) like ChatGPT could have on society, including fake news, plagiarism, and social security issues. To address these concerns, a team of researchers led by Biyang Guo conducted an extensive study comparing ChatGPT's responses with those of human experts across various domains such as open-domain, financial, medical, legal, and psychological areas. The team collected tens of thousands of comparison responses from both sources and created the Human ChatGPT Comparison Corpus (HC3) dataset for analysis. The study revealed interesting insights into the characteristics of ChatGPT's responses compared to those generated by humans. The researchers conducted comprehensive evaluations and linguistic analyses to identify differences and gaps between the two sources. They also explored future directions for LLMs based on their findings. In addition to this analysis, the team also developed three different detection systems to effectively distinguish between text generated by ChatGPT or humans. They explored several key factors that influence their effectiveness and evaluated them in different scenarios. Overall, this study provides valuable insights into the capabilities and limitations of large language models like ChatGPT. The HC3 dataset is publicly available along with code and models at https://github.com/Hello-SimpleAI/chatgpt-comparison-detection for further research in this area.