# Survey of Hallucination in Natural Language Generation

**Authors:** Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, Pascale Fung  
**arXiv:** [2202.03629](https://arxiv.org/abs/2202.03629)  
**Date:** 2022-02-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2202.03629v1/

## Key points

- Natural Language Generation (NLG) has made significant advancements in recent years due to the development of deep learning technologies such as Transformer-based language models.
- NLG enables more fluent and coherent natural language generation for downstream tasks like abstractive summarization, dialogue generation, and data-to-text generation.
- However, NLG also generates hallucinated texts that fail to meet users' expectations in real-world scenarios.
- Researchers have presented various evaluation and mitigation methods for hallucinations in different tasks.
- Ziwei Ji and colleagues from Hong Kong University of Science and Technology provide an overview of research progress and challenges related to the hallucination problem in NLG.
- The authors categorize hallucinations into intrinsic (generated text contradicts factual information or common sense) and extrinsic (generated text does not relate to the input).
- They discuss contributors to hallucinations such as data quality issues or errors during training or inference.
- Statistical metrics based on perplexity scores or n-gram overlap ratios as well as model-based metrics like adversarial attacks or gradient-based saliency maps are proposed for evaluating hallucination severity.
- Human evaluation methods like crowd-sourcing are also discussed.
- Various mitigation methods including data augmentation or filtering and regularization or fine-tuning are presented for reducing hallucinations in NLG systems.
- The authors emphasize the need for a standardized evaluation framework for future directions in metrics and mitigation methods.

## Summary

Natural Language Generation (NLG) has made significant advancements in recent years due to the development of deep learning technologies such as Transformer-based language models. This progress has enabled more fluent and coherent natural language generation for downstream tasks like abstractive summarization, dialogue generation, and data-to-text generation. However, NLG also generates hallucinated texts that fail to meet users' expectations in real-world scenarios. To address this issue, researchers have presented various evaluation and mitigation methods for hallucinations in different tasks. In this survey paper, Ziwei Ji and colleagues from Hong Kong University of Science and Technology provide an overview of research progress and challenges related to the hallucination problem in NLG. The authors categorize hallucinations into intrinsic (generated text contradicts factual information or common sense) and extrinsic (generated text does not relate to the input). They discuss contributors to hallucinations such as data quality issues or errors during training or inference. Statistical metrics based on perplexity scores or n-gram overlap ratios as well as model-based metrics like adversarial attacks or gradient-based saliency maps are proposed for evaluating hallucination severity. Human evaluation methods like crowd-sourcing are also discussed. Various mitigation methods including data augmentation or filtering and regularization or fine-tuning are presented for reducing hallucinations in NLG systems. The authors emphasize the need for a standardized evaluation framework for future directions in metrics and mitigation methods. In summary, this survey paper provides a comprehensive overview of research progress related to the hallucination problem in NLG with proposed metrics and mitigation methods that could facilitate collaborative efforts among researchers working on different downstream tasks.