# GPTEval: NLG Evaluation using GPT-4 with Better Human Alignment

**Authors:** Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, Chenguang Zhu  
**arXiv:** [2303.16634](https://arxiv.org/abs/2303.16634)  
**Date:** 2023-03-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.16634v1/

## Key points

- The paper addresses the challenge of automatically measuring the quality of texts generated by NLG systems.
- Traditional reference-based metrics like BLEU and ROUGE have limited correlation with human judgments, especially for creative and diverse tasks.
- LLM-based evaluators have lower human correspondence compared to medium-size neural evaluators.
- GPTEval is introduced as a framework that utilizes large language models with chain-of-thoughts (CoT) and a form-filling paradigm for NLG evaluation.
- GPTEval achieves a high Spearman correlation of 0.514 with human judgments in the text summarization task, outperforming previous methods significantly.
- LLM-based evaluators may exhibit bias towards LLM-generated texts, highlighting the need for further investigation into mitigating biases in evaluation methods.
- GPTEval contributes to advancing NLG evaluation by leveraging large language models to improve alignment with human judgments.
- The findings have implications for enhancing the assessment of NLG system outputs in domains requiring creativity and diversity.

## Summary

The paper titled "GPTEval: NLG Evaluation using GPT-4 with Better Human Alignment" addresses the challenge of automatically measuring the quality of texts generated by natural language generation (NLG) systems. Traditional reference-based metrics like BLEU and ROUGE have shown limited correlation with human judgments, particularly for tasks that require creativity and diversity. Recent studies propose using large language models (LLMs) as reference-free metrics for NLG evaluation, which can be applied to new tasks lacking human references. However, LLM-based evaluators still exhibit lower human correspondence compared to medium-size neural evaluators. To overcome these limitations, the authors introduce GPTEval, a framework that utilizes large language models with chain-of-thoughts (CoT) and a form-filling paradigm to assess the quality of NLG outputs. The framework is evaluated on two generation tasks: text summarization and dialogue generation. The results demonstrate that GPTEval with GPT-4 as the backbone model achieves a high Spearman correlation of 0.514 with human judgments in the summarization task, outperforming previous methods significantly. Additionally, the paper presents an analysis of LLM-based evaluators' behavior and highlights an issue where they may exhibit bias towards LLM-generated texts. This finding emphasizes the need for further investigation into mitigating biases in LLM-based evaluation methods. Overall, this research contributes to advancing NLG evaluation by introducing GPTEval as an effective framework that leverages large language models to improve alignment with human judgments. The findings have implications for enhancing the assessment of NLG system outputs in various domains requiring creativity and diversity.