# A Survey of Evaluation Metrics Used for NLG Systems

**Authors:** Ananya B. Sai, Akash Kumar Mohankumar, Mitesh M. Khapra  
**arXiv:** [2008.12009](https://arxiv.org/abs/2008.12009)  
**Date:** 2020-08-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2008.12009v2/

## Key points

- Deep Learning has led to increased interest in Natural Language Generation (NLG) tasks, including image captioning.
- Automatic evaluation metrics are needed to track advancements in NLG systems.
- Early heuristic-based metrics like BLEU and ROUGE are inadequate for capturing nuances in NLG tasks.
- Since 2014, there has been a surge in proposed evaluation metrics due to the increasing number of NLG models and limitations of current metrics.
- There is a shift from using predetermined heuristic formulas to trained transformer models for evaluation.
- This survey highlights challenges in automatically evaluating NLG systems and provides a taxonomy of existing evaluation metrics.
- The survey describes various metrics, their contributions, shortcomings, and the methodology used for evaluation.
- Suggestions and recommendations are offered for improving automatic evaluation metrics.
- The survey aims to help researchers quickly familiarize themselves with developments in NLG evaluation.

## Summary

The success of Deep Learning has sparked a growing interest in various Natural Language Generation (NLG) tasks, including image captioning. As NLG progresses rapidly, there is a need for accurate automatic evaluation metrics to track advancements in the field. However, evaluating NLG systems automatically presents significant challenges. Early heuristic-based metrics like BLEU and ROUGE have proven inadequate for capturing the nuances of different NLG tasks. The increasing number of NLG models and the limitations of current metrics have led to a surge in proposed evaluation metrics since 2014. Additionally, there has been a shift from using predetermined heuristic formulas to trained transformer models for evaluation. To address these developments and assist researchers, this survey highlights the challenges involved in automatically evaluating NLG systems. It provides a coherent taxonomy of existing evaluation metrics to better understand the field's progress. The survey describes various metrics in detail, emphasizing their key contributions as well as main shortcomings identified in current metrics and outlines the methodology used to evaluate these evaluation metrics. Finally, the survey offers suggestions and recommendations for improving automatic evaluation metrics going forward. By providing an overview of existing NLG metrics and their advancements over recent years, this survey aims to help both new and established researchers quickly familiarize themselves with developments in NLG evaluation.