# TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models

**Authors:** Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, Idan Szpektor  
**arXiv:** [2305.11171](https://arxiv.org/abs/2305.11171)  
**Date:** 2023-05-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.11171v1/

## Key points

- Generative summarization models often produce summaries that are factually inconsistent with their input documents.
- Natural Language Inference (NLI) models are commonly used to evaluate factual consistency, but have limited success due to the lack of entailment phenomena in NLI datasets.
- Previous work improved NLI models with synthetic training data generated by perturbing human-written summaries, but this approach has limitations in coverage and style differences from real model-generated summaries.
- Large language models (LLMs) have shown promising results in directly evaluating generative tasks such as factual consistency in summarization, but they are too computationally expensive for practical use.
- TrueTeacher is introduced as a method for generating synthetic data by annotating diverse model-generated summaries using a LLM.
- TrueTeacher does not rely on human-written summaries and is multilingual by nature.
- Experiments on the TRUE benchmark show that a student model trained using TrueTeacher data substantially outperforms both state-of-the-art models with similar capacity and the LLM teacher.
- A systematic study demonstrates TrueTeacher's superiority and robustness to domain-shift compared to existing synthetic data generation methods.
- The method also generalizes to multilingual scenarios using the mFACE dataset.
- Finally, a large-scale synthetic dataset with 1.4M examples generated using TrueTeacher is released, highlighting the usefulness of this method for improving factual consistency evaluation in summarization models.

## Summary

Generative summarization models often generate summaries that are factually inconsistent with their input documents. To evaluate factual consistency, Natural Language Inference (NLI) models are commonly used but have limited success due to the lack of entailment phenomena in NLI datasets. Previous work improved NLI models with synthetic training data generated by perturbing human-written summaries, but this approach has limitations in coverage and style differences from real model-generated summaries. Large language models (LLMs) have shown promising results in directly evaluating generative tasks such as factual consistency in summarization; however, they are too computationally expensive for practical use. To address these limitations, TrueTeacher is introduced as a method for generating synthetic data by annotating diverse model-generated summaries using a LLM. Unlike prior work, TrueTeacher does not rely on human-written summaries and is multilingual by nature. Experiments on the TRUE benchmark show that a student model trained using TrueTeacher data substantially outperforms both state-of-the-art models with similar capacity and the LLM teacher. A systematic study demonstrates TrueTeacher's superiority and robustness to domain-shift compared to existing synthetic data generation methods. The method also generalizes to multilingual scenarios using the mFACE dataset. Finally, a large-scale synthetic dataset with 1.4M examples generated using TrueTeacher is released, highlighting the usefulness of this method for improving factual consistency evaluation in summarization models.