# TruthfulQA: Measuring How Models Mimic Human Falsehoods

**Authors:** Stephanie Lin, Jacob Hilton, Owain Evans  
**arXiv:** [2109.07958](https://arxiv.org/abs/2109.07958)  
**Date:** 2021-09-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2109.07958v1/

## Key points

- Authors Stephanie Lin, Jacob Hilton, and Owain Evans introduce a benchmark to evaluate truthfulness of language models
- Benchmark consists of 817 questions across 38 categories designed to elicit false answers
- Study tested models including GPT-3, GPT-Neo/J, GPT-2, and T5-based model
- Best-performing model was truthful in only 58% of cases, lower than human performance at 94%
- Larger models tended to be less truthful overall; e.g., GPT-J exhibited a 17% decrease in truthfulness compared to smaller counterpart
- Models often generated false answers mirroring common misconceptions found in human texts
- Concerns raised about potential for models to deceive individuals with inaccurate information
- Model performance typically improves with increased size but not for truthfulness
- Authors recommend exploring alternative training objectives beyond imitation of web text for improving truthfulness

## Summary

In their paper titled "TruthfulQA: Measuring How Models Mimic Human Falsehoods," authors Stephanie Lin, Jacob Hilton, and Owain Evans introduce a benchmark designed to evaluate the truthfulness of language models in generating answers to questions. The benchmark consists of 817 questions across 38 categories such as health, law, finance, and politics. These questions were specifically crafted to elicit false answers that some humans might provide due to misconceptions or false beliefs. The study tested several prominent language models including GPT-3, GPT-Neo/J, GPT-2, and a T5-based model. The results revealed that the best-performing model was truthful in only 58% of cases, significantly lower than human performance at 94%. Interestingly, the larger models tended to be less truthful overall. For instance, the 6B-parameter GPT-J model exhibited a 17% decrease in truthfulness compared to its smaller 125M-parameter counterpart. The authors observed that these models often generated false answers that mirrored common misconceptions found in human texts. This raises concerns about the potential for these models to deceive individuals by providing inaccurate information. The study also highlighted a notable difference from other natural language processing tasks where model performance typically improves with increased size. Based on their findings, the authors suggest that simply scaling up models may not be sufficient for enhancing truthfulness. They recommend exploring alternative training objectives beyond imitation of text from the web as a more effective approach for improving truthfulness in language models. The research provides valuable insights into the challenges associated with ensuring accuracy and reliability in AI-generated responses to questions across various domains.