# BARTScore: Evaluating Generated Text as Text Generation

**Authors:** Weizhe Yuan, Graham Neubig, Pengfei Liu  
**arXiv:** [2106.11520](https://arxiv.org/abs/2106.11520)  
**Date:** 2021-06-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2106.11520v1/

## Key points

- Authors focus on evaluating the quality of generated text in NLP applications
- Introduce BARTScore, a method that uses pre-trained models to evaluate fluency, accuracy, and effectiveness of generated texts
- BARTScore outperforms existing metrics in 16 out of 22 test settings across 16 datasets and 7 perspectives
- BARTScore performs better than other metrics in most settings for text summarization tasks
- Fine-tuning tasks can improve BARTScore's performance on some datasets but not others
- BARTScore performs well in evaluating factuality compared to other metrics like FactCC and QAGS when using CNN as a fine-tuning task
- Human evaluation is important for assessing text quality from different perspectives
- Existing evaluation metrics have limitations, while BARTScore aims to provide a comprehensive approach
- Code and interactive leaderboard are provided for researchers to evaluate different metrics.

## Summary

In this work, the authors focus on evaluating the quality of generated text in various natural language processing (NLP) applications such as machine translation, summarization, and dialog. They propose a method called BARTScore which utilizes pre-trained sequence-to-sequence models to evaluate the fluency, accuracy and effectiveness of generated texts. The authors use BART, an encoder-decoder based pre-trained model to develop the BARTScore metric. They demonstrate that BARTScore outperforms existing metrics in 16 out of 22 test settings across 16 datasets and 7 different perspectives. The authors specifically analyze the performance of BARTScore in text summarization tasks. They compare it with other metrics such as BERTScore and MoverScore and find that vanilla BARTScore performs significantly better on most settings except for the informativeness perspective on SummEval dataset. They also observe that introducing fine-tuning tasks can further improve the performance on some datasets but not on others. Additionally, they analyze how well BARTScore performs in evaluating factuality in short generated summaries compared to human baselines and other factuality metrics like FactCC and QAGS. They find that using CNN as a fine-tuning task improves BARTScore's performance significantly compared to other metrics. The authors discuss the importance of human evaluation as a gold-standard method for assessing text quality from different perspectives such as informativeness, relevance, fluency, coherence, factuality, semantic coverage and adequacy. Existing evaluation metrics were designed to cover only a subset of these perspectives or require separate judgments for each type. The proposed BARTScore metric aims to address these limitations by providing a comprehensive evaluation approach. Overall, this work presents an effective method for evaluating generated text using pre-trained models like BART and introduces the BARTScore metric that outperforms existing top-scoring metrics in various NLP applications. The authors provide code to calculate BARTScore and an interactive leaderboard for meta-evaluation allowing researchers to understand the strengths weaknesses and complementarity of different evaluation metrics.