# Learning to summarize from human feedback

**Authors:** Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano  
**arXiv:** [2009.01325](https://arxiv.org/abs/2009.01325)  
**Date:** 2020-09-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2009.01325v1/

## Key points

- Authors address challenges faced by language models in training and evaluation due to limitations of existing data and metrics
- Proposed novel approach involves training a model to optimize for human preferences by collecting a large dataset of human comparisons between summaries
- Model is then used as a reward function to fine-tune summarization policy through reinforcement learning
- Study focuses on TL;DR dataset of Reddit posts and shows significant improvements in summary quality compared to human reference summaries and larger models fine-tuned with supervised learning alone
- Improvements transfer effectively to CNN/DM news articles without news-specific fine-tuning
- Extensive analyses conducted on human feedback dataset and fine-tuned models to understand performance better
- Reward model generalizes well to new datasets and leads to superior summaries compared to optimizing ROUGE based on human evaluations
- Emphasizes importance of optimizing for desired outcomes rather than relying solely on traditional metrics

## Summary

In their paper titled "Learning to Summarize from Human Feedback," authors Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei and Paul Christiano address the challenges faced by language models in training and evaluation due to limitations of existing data and metrics. Summarization models are typically trained on predicting human reference summaries and evaluated using metrics like ROUGE; however, these measures often fall short in capturing true summary quality. To overcome this limitation, the authors propose a novel approach where a model is trained to optimize for human preferences. This involves collecting a large dataset of human comparisons between summaries to train a model that can predict preferred summaries. The resulting model is then used as a reward function to fine-tune a summarization policy through reinforcement learning. The study focuses on the TL;DR dataset of Reddit posts and demonstrates significant improvements in summary quality compared to both human reference summaries and larger models fine-tuned with supervised learning alone. Remarkably, these improvements also transfer effectively to CNN/DM news articles without any news-specific fine-tuning. Extensive analyses are conducted on the human feedback dataset and the fine-tuned models to better understand their performance. The authors establish that their reward model generalizes well to new datasets and leads to superior summaries compared to optimizing ROUGE based on human evaluations. Overall, the findings presented in this paper urge machine learning researchers to consider how their training loss impacts the actual behavior of their models. This emphasizes the importance of optimizing for desired outcomes rather than relying solely on traditional metrics.