# SummEval: Re-evaluating Summarization Evaluation

**Authors:** Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Richard Socher, Dragomir Radev  
**arXiv:** [2007.12626](https://arxiv.org/abs/2007.12626)  
**Date:** 2020-07-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2007.12626v1/

## Key points

- Scarcity of comprehensive and up-to-date studies on evaluation metrics for text summarization
- Lack of consensus regarding evaluation protocols
- Proposal of five dimensions to improve summarization evaluation methods
- Re-evaluation of 12 automatic evaluation metrics using neural summarization model outputs alongside expert and crowd-sourced human annotations
- Benchmarking 23 recent summarization models using automatic evaluation metrics for reliable comparison
- Assembly and sharing of the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset in a unified format
- Development and sharing of a toolkit with an extensible and unified API for evaluating summarization models across various automatic metrics
- Compilation of the largest and most diverse collection of human judgments on model-generated summaries from the CNN/Daily Mail dataset, annotated by both expert judges and crowd source workers
- Contribution to promoting a more complete evaluation protocol for text summarization
- Advancement in research for developing evaluation metrics that better correlate with human judgments.

## Summary

In their paper titled "SummEval: Re-evaluating Summarization Evaluation," authors Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Richard Socher, and Dragomir Radev highlight the scarcity of comprehensive and up-to-date studies on evaluation metrics for text summarization. They also point out the lack of consensus regarding evaluation protocols which hinders progress in this field. To address these shortcomings, the authors propose five dimensions to improve summarization evaluation methods. Firstly, they re-evaluate 12 automatic evaluation metrics using neural summarization model outputs alongside expert and crowd-sourced human annotations. This comprehensive and consistent approach allows for a more accurate assessment of these metrics. Secondly, the authors benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics. By doing so they provide a reliable comparison between different models and their performance. Thirdly, they assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format. This dataset serves as a valuable resource for researchers working on text summarization tasks. Fourthly, the authors develop and share a toolkit that offers an extensible and unified API for evaluating summarization models across various automatic metrics. This toolkit enhances accessibility and facilitates further research in this area. Lastly, they compile the largest and most diverse collection of human judgments on model-generated summaries from the CNN/Daily Mail dataset. These judgments are annotated by both expert judges and crowd source workers providing valuable insights into how well these model-generated summaries align with human judgments. The authors hope that their work will contribute to promoting a more complete evaluation protocol for text summarization while advancing research in developing evaluation metrics that better correlate with human judgments.