# Prover-Verifier Games improve legibility of LLM outputs

**Authors:** Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, Yuri Burda  
**arXiv:** [2407.13692](https://arxiv.org/abs/2407.13692)  
**Date:** 2024-07-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.13692v1/

## Key points

- Authors explore the concept of legibility in Large Language Models (LLMs) by focusing on solving grade-school math problems
- Importance of clear and easily verifiable reasoning to enhance confidence in LLM outputs is highlighted
- Solely optimizing solutions for correctness can compromise legibility
- Introduction of a novel training algorithm inspired by the Prover-Verifier Game proposed by Anil et al. in 2021
- Algorithm involves training small verifiers, "helpful" provers, and "sneaky" provers to improve solution accuracy and resilience against adversarial attacks
- Improvement observed in helpful prover's accuracy and verifier's resilience over the training period
- Training for legibility transfers effectively to humans verifying solution correctness under time constraints
- Human accuracy increases when validating solutions from helpful prover but decreases when assessing those from sneaky prover
- Training with small verifiers enhances output legibility in large LLMs for human comprehension
- Leveraging legibility training against small verifiers improves alignment between superhuman models and human understanding

## Summary

In their paper titled "Prover-Verifier Games Improve Legibility of LLM Outputs," authors Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda explore the concept of legibility in Large Language Models (LLMs) by focusing on solving grade-school math problems. They highlight the importance of clear and easily verifiable reasoning to enhance confidence in LLM outputs. The study reveals that solely optimizing solutions for correctness can compromise legibility. To address this issue, the authors introduce a novel training algorithm inspired by the Prover-Verifier Game proposed by Anil et al. in 2021.

 are used to improve  in . The authors focus on solving grade-school math problems and emphasize the importance of clear and easily verifiable reasoning to enhance confidence in LLM outputs. However, solely optimizing solutions for correctness can compromise legibility. To address this issue, they introduce a novel  inspired by the Prover-Verifier Game proposed by Anil et al. in 2021. The algorithm involves iteratively training small verifiers to assess solution correctness, "helpful" provers to generate accurate solutions accepted by the verifier, and "sneaky" provers to produce incorrect solutions that deceive the verifier. Through their experiments, the authors observe an improvement in the helpful prover's accuracy and the verifier's resilience against adversarial attacks over the training period. Moreover, they demonstrate that training for legibility transfers effectively to humans tasked with verifying solution correctness under time constraints. Throughout LLM training, human accuracy increases when validating solutions from the helpful prover but decreases when assessing those from the sneaky prover. This underscores the efficacy of training with small verifiers to enhance output legibility in large LLMs for human comprehension. The results suggest that leveraging legibility training against small verifiers presents a practical approach to improving alignment between superhuman models and human understanding. By emphasizing clear reasoning and easy verification processes, this research contributes valuable insights towards enhancing confidence and interpretability in LLM outputs.