# What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation

**Authors:** Dingyi Yang, Qin Jin  
**arXiv:** [2408.14622](https://arxiv.org/abs/2408.14622)  
**Date:** 2024-08-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2408.14622v1/

## Key points

- Authors Dingyi Yang and Qin Jin explore story evaluation in the age of artificial intelligence
- Advancements in Large Language Models (LLMs) have led to an increase in both quantity and quality of automatically generated stories
- Automatic story evaluation methods are necessary to assess generative capabilities of computing systems and compare machine-generated narratives with human-crafted ones
- Evaluating a story presents unique challenges compared to other generation evaluation tasks like machine translation, requiring a nuanced approach focusing on coherence, character development, and interestingness
- The authors review existing research on storytelling tasks, including text-to-text, visual-to-text, and text-to-visual scenarios
- Various human criteria are identified for measuring stories along with benchmark datasets for evaluation purposes
- A taxonomy is proposed by the authors to categorize existing metrics for story assessment and suggest new ones that can be adopted
- The survey discusses the potential for human-AI collaboration in story evaluation and generation processes
- Future research directions include expanding evaluations across different domains to enhance computational storytelling capabilities

## Summary

In their comprehensive survey titled "What Makes a Good Story and How Can We Measure It? ", authors Dingyi Yang and Qin Jin delve into the realm of story evaluation in the age of artificial intelligence. They highlight the significant increase in both quantity and quality of automatically generated stories, thanks to advancements in Large Language Models (LLMs). This surge has necessitated the development of automatic story evaluation methods to assess the generative capabilities of computing systems and compare the quality of machine-generated narratives with those crafted by humans. The authors point out that evaluating a story poses unique challenges compared to other generation evaluation tasks like machine translation. While fluency and accuracy are key metrics in translation tasks, assessing a story's overall coherence, character development, and interestingness requires a more nuanced approach. To address these complexities, Yang and Jin conduct a thorough review of existing research on storytelling tasks, including text-to-text, visual-to-text, and text-to-visual scenarios. They also identify various human criteria used to measure stories and present benchmark datasets for evaluation purposes. To organize the plethora of evaluation metrics available for story assessment, the authors propose a taxonomy that categorizes existing metrics and suggests new ones that can be adopted. They provide detailed descriptions of these metrics along with discussions on their strengths and limitations. Additionally, they explore the potential for human-AI collaboration in both story evaluation and generation processes. Looking towards future research directions, Yang and Jin suggest expanding from story evaluation to encompass general evaluations across different domains. By shedding light on the intricacies of evaluating narrative content in an AI-driven landscape, this survey serves as a valuable resource for researchers seeking to enhance computational storytelling capabilities.