# FABLES: Evaluating faithfulness and content selection in book-length summarization

**Authors:** Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, Mohit Iyyer  
**arXiv:** [2404.01261](https://arxiv.org/abs/2404.01261)  
**Date:** 2024-04-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.01261v1/

## Key points

- Authors conducted a large-scale human evaluation on summaries of books published in 2023 or 2024
- Created the FABLES dataset at a cost of $5.2K USD
- Ranked LLM summarizers based on faithfulness, with Claude-3-Opus outperforming closed-source LLMs significantly
- Most unfaithful claims in the summaries related to events and character states, requiring indirect reasoning to invalidate
- LLM-based auto-raters for factuality and coherence did not strongly correlate with human annotations when detecting unfaithful claims
- Importance of detecting unfaithful claims highlighted as a crucial future direction for summarization evaluation
- Explored content selection errors by developing a typology of omission errors related to crucial narrative elements
- Systematic over-emphasis on events occurring towards the end of the book

## Summary

, , , , 


In their paper titled "FABLES: Evaluating Faithfulness and Content Selection in Book-Length Summarization," authors Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer address the challenges of evaluating faithfulness and content selection in summaries generated by long-context large language models (LLMs) for fictional books. The authors conducted a large-scale human evaluation focusing on summaries of books published in 2023 or 2024 to mitigate data contamination issues. The study involved annotators who had fully read each book prior to the annotation task to ensure accuracy. The authors collected annotations on 3,158 claims made in LLM-generated summaries of 26 books, resulting in the creation of the FABLES dataset at a cost of $5.2K USD. This dataset allowed them to rank LLM summarizers based on faithfulness, with Claude-3-Opus outperforming closed-source LLMs significantly and the open-source Mixtral performing on par with GPT-3.5-Turbo. Analysis of the annotations revealed that most unfaithful claims in the summaries related to events and character states, often requiring indirect reasoning over the narrative to invalidate. Despite the reliability of LLM-based auto-raters for factuality and coherence in other contexts, several LLM raters of faithfulness implemented by the authors did not correlate strongly with human annotations when detecting unfaithful claims. The experiments conducted by the authors highlighted the importance of detecting unfaithful claims as a crucial future direction for summarization evaluation and as a testbed for long-context understanding. Additionally, they explored content selection errors in book-length summarization by developing a typology of omission errors related to crucial narrative elements. They also identified a systematic over-emphasis on events occurring towards the end of the book. Overall, this research sheds light on the challenges and opportunities in evaluating faithfulness and content selection in LLM-generated summaries of fictional books, providing valuable insights for future advancements in summarization technology.