# Evaluating Text Summaries Generated by Large Language Models Using OpenAI's GPT

**Authors:** Hassan Shakil, Atqiya Munawara Mahi, Phuoc Nguyen, Zeydy Ortiz, Mamoun T. Mardini  
**arXiv:** [2405.04053](https://arxiv.org/abs/2405.04053)  
**Date:** 2024-05-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.04053v1/

## Key points

- Text summarization in NLP is crucial for information management in various domains such as news reporting, report generation, and conversational analysis.
- Evolution of text summarization from rule-based systems to sophisticated machine learning strategies driven by the limitations of early approaches in capturing natural language nuances.
- Transformer models like BERT, BART, T5, PEGASUS, ProphetNet have significantly advanced text summarization capabilities through self-supervised training and novel pretraining objectives.
- DistilBART exemplifies knowledge distillation techniques for deploying large transformer models in resource-constrained environments without compromising performance.
- Study evaluates text summaries generated by transformer models using OpenAI's GPT as an independent evaluator, showcasing significant correlations between GPT evaluations and traditional metrics like ROUGE and LSA.

## Summary

, , , , 

This research delves into the realm of text summarization within Natural Language Processing (NLP), a crucial aspect of information management across various domains such as news reporting, report generation, and conversational analysis. Initially rooted in rule-based systems pioneered by Luhn [1958], text summarization has evolved from simplistic heuristics to more sophisticated machine learning strategies. This evolution was driven by the limitations of early approaches in capturing the nuances of natural language. Early machine learning techniques like Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units laid the foundation for extractive summarization models, addressing challenges related to temporal dependencies within text sequences. The advent of transformer models, notably introduced by Vaswani et al. [2017], revolutionized NLP with their self-attention mechanism, enabling a more comprehensive understanding of contextual relationships in text. Models like BERT by Devlin et al. [2018] further advanced text representation through self-supervised training on extensive corpora. Subsequent innovations led to transformer models like BART and T5, renowned for their exceptional performance in summarization tasks due to robust architecture and training methodologies. Further enhancements in transformer-based models include PEGASUS and ProphetNet, which introduced novel pretraining objectives to bolster summarization capabilities. DistilBART exemplifies knowledge distillation techniques that enable the deployment of large transformer models in resource-constrained environments without compromising performance. Building upon this foundation, this study evaluates text summaries generated by leading transformer models using OpenAI's GPT as an independent evaluator. By employing traditional metrics such as ROUGE and Latent Semantic Analysis (LSA) alongside innovative AI-driven evaluations, the research explores GPT's effectiveness in enhancing automated text summarization quality. The findings showcase significant correlations between GPT evaluations and traditional metrics, particularly in assessing relevance and coherence of summaries. Overall, this research highlights GPT's potential as a robust tool for evaluating text summaries, offering valuable insights that complement established metrics and pave the way for comparative analysis of transformer-based models in NLP tasks. The study underscores the practical application of AI tools in processing vast amounts of information efficiently and effectively.