# Comparing Abstractive Summaries Generated by ChatGPT to Real Summaries Through Blinded Reviewers and Text Classification Algorithms

**Authors:** Mayank Soni, Vincent Wade  
**arXiv:** [2303.17650](https://arxiv.org/abs/2303.17650)  
**Date:** 2023-03-30  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2303.17650v3/

## Key points

- Large Language Models (LLMs) have gained significant attention for their exceptional performance across various tasks.
- ChatGPT, developed by OpenAI, is a recent addition to the LLM family and has been hailed as a disruptive technology due to its human-like text generation capabilities.
- A study focused on evaluating ChatGPT's performance in Abstractive Summarization using automated metrics and blinded human reviewers.
- Limitations of the study included comparing only 50 summaries, not exploring different prompts for generating summaries, lack of comparison with other models or baselines, reliance on native English-speaking reviewers, and potential for improving automatic summary detection accuracy through more advanced algorithms.
- Text classification algorithms could differentiate between real and generated summaries, but human reviewers struggled to distinguish between them due to intentional selection of prompts closely resembling original summaries.
- The study achieved a 90% accuracy rate in identifying ChatGPT-generated summaries.
- Previous studies have evaluated ChatGPT's performance in various tasks such as machine translation and medical examinations, showing competitive results in some areas but limitations in others.
- Interactive approaches in summarization have shown improvements in ROUGE scores while revealing both believable outputs and detectable differences by AI tools and skeptical human reviewers when compared with original content.

## Summary

Large Language Models (LLMs) have gained significant attention for their exceptional performance across various tasks. ChatGPT, developed by OpenAI, is a recent addition to the LLM family and has been hailed as a disruptive technology due to its human-like text generation capabilities. While anecdotal examples on the internet have highlighted both the strengths and weaknesses of ChatGPT, there is a limited amount of systematic research available. To contribute to the existing literature on ChatGPT, this study focused on evaluating its performance in Abstractive Summarization using automated metrics and blinded human reviewers. The study had limitations, including a restriction on comparing only 50 summaries, not exploring different prompts for generating summaries, lack of comparison with other models or baselines, reliance on native English-speaking reviewers, and potential for improving automatic summary detection accuracy through more advanced algorithms. The results revealed that while text classification algorithms could differentiate between real and generated summaries, human reviewers struggled to distinguish between them. Reviewers were uncertain about whether a summary was produced by ChatGPT or a human writer. This difficulty was attributed to the lack of distinguishing features between the two sources, which was intentional in selecting prompts that closely resembled original summaries. Additionally, the study achieved a 90% accuracy rate in identifying ChatGPT-generated summaries. In related work, previous studies have evaluated ChatGPT's performance in various tasks such as machine translation and medical examinations. These evaluations have shown competitive results in some areas but also highlighted limitations in others. In summarization specifically, interactive approaches have shown improvements in ROUGE scores, while comparisons with original content have revealed both believable outputs and detectable differences by AI tools and skeptical human reviewers. Summarization itself involves shortening large texts while preserving key information through extractive or abstractive methods. The study utilized a specific dataset for evaluation purposes and identified areas for future research improvement such as exploring different prompts for generating summaries and comparing ChatGPT's performance with other models. Overall, this study contributes valuable insights into ChatGPT's capabilities in abstractive summarization and highlights the challenges faced by both automated algorithms and human reviewers in distinguishing between machine-generated content and human-written text.