# Can Large Language Models Be an Alternative to Human Evaluations?

**Authors:** Cheng-Han Chiang, Hung-yi Lee  
**arXiv:** [2305.01937](https://arxiv.org/abs/2305.01937)  
**Date:** 2023-05-03  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2305.01937v1/

## Key points

- Large language models (LLMs) can be used as an alternative to human evaluation for assessing text quality
- LLMs are given task instructions, samples, and questions used in human evaluation to generate responses
- LLM evaluation is compared with expert human evaluation in open-ended story generation and adversarial attacks tasks
- Results of LLM evaluation are consistent with human evaluation, indicating effective text quality assessment
- LLM evaluation results are stable across different formatting of task instructions and sampling algorithms
- This study demonstrates the potential of using LLMs for text quality assessment and discusses limitations and ethical considerations
- A detailed example is provided on using LLM evaluation in open-ended story generation with the WritingPrompts dataset
- LLMs offer reproducibility and stability compared to traditional human evaluation methods in NLP tasks.

## Summary

This paper explores the potential of using large language models (LLMs) as an alternative to human evaluation for assessing the quality of texts generated by machine learning models or written by humans. The authors present LLMs with task instructions, samples to be evaluated, and questions used in human evaluation and ask the LLMs to generate responses to those questions. This process is called LLM evaluation. The authors compare the results of LLM evaluation with expert human evaluation in two natural language processing tasks: open-ended story generation and adversarial attacks. They find that the results of LLM evaluation are consistent with human evaluation, indicating that LLMs can effectively assess text quality. The authors also demonstrate that the results of LLM evaluation are stable across different formatting of task instructions and sampling algorithms. This study is the first to show the potential of using LLMs for text quality assessment and discusses its limitations and ethical considerations associated with it. Additionally, a detailed example is provided on how to use LLM evaluation in open-ended story generation using the WritingPrompts dataset. Overall, this research highlights how LLMs can be a valuable tool for evaluating text quality in NLP tasks offering reproducibility and stability compared to traditional human evaluation methods.