# Fine-tuning Language Models for Factuality

**Authors:** Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D. Manning, Chelsea Finn  
**arXiv:** [2311.08401](https://arxiv.org/abs/2311.08401)  
**Date:** 2023-11-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.08401v1/

## Key points

- Manual fact-checking is time-consuming and expensive
- Authors propose a method to fine-tune language models for improved factuality without human labeling
- Two innovations in NLP are leveraged: measuring factuality using external knowledge base or model's confidence scores, and direct preference optimization algorithm
- Factuality preference rankings are generated using retrieval systems or novel retrieval-free approach
- Learning from these rankings enhances factuality of Llama-2 on various topics compared to other strategies
- Significant reduction in factual error rates for biographies and medical questions compared to Llama-2-chat (58% reduction for biographies, 40% reduction for medical questions)
- Future research directions include combining factuality tuning with existing methods, scaling up approach to larger models, more benchmarks on long-form language model generations' factuality, integrating reinforcement learning with factuality rankings
- Work presents practical and effective strategy for improving language models' ability to generate factual content in long-form settings.

## Summary

Manual fact-checking is time-consuming and expensive, making it challenging to acquire human factuality labels. To tackle this problem, the authors propose a method to fine-tune language models for improved factuality without relying on human labeling. The authors leverage two recent innovations in natural language processing (NLP) to achieve their goal. First, they utilize methods that measure the factuality of open-ended text by assessing consistency with an external knowledge base or a large model's confidence scores. Second, they employ the direct preference optimization algorithm, which allows for straightforward fine-tuning of language models based on preference rankings rather than supervised imitation. To evaluate their approach, the authors generate factuality preference rankings using existing retrieval systems or their novel retrieval-free approach. They demonstrate that learning from these automatically generated rankings significantly enhances the factuality of Llama-2 on various topics compared to other strategies targeted at improving factuality. The results show a substantial reduction in factual error rates when generating biographies and answering medical questions using Llama-2 at a 7B scale. Specifically, there is a 58% reduction in factual error rate for biographies and a 40% reduction for medical questions compared to Llama-2-chat. The authors conclude by highlighting future research directions such as exploring additional ways to combine factuality tuning with existing methods and scaling up their approach to larger models. They also emphasize the need for more benchmarks on long-form language model generations' factuality and suggest investigating how best to integrate typical rewards and approaches from reinforcement learning with factuality rankings. Overall, this work presents a practical and effective strategy for improving language models' ability to generate factual content in long-form settings.