# Training language models to follow instructions with human feedback

**Authors:** Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe  
**arXiv:** [2203.02155](https://arxiv.org/abs/2203.02155)  
**Date:** 2022-03-04  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2203.02155v1/

## Key points

- Simply making language models bigger does not necessarily make them better at following a user's intent.
- Large language models can generate outputs that are untruthful, toxic, or not helpful to the user.
- The authors propose an approach for aligning language models with user intent on various tasks by fine-tuning them with human feedback.
- They collect a dataset of labeler demonstrations of desired model behavior and use it to fine-tune GPT-3 using supervised learning.
- They also collect a dataset of rankings of model outputs and further fine-tune this supervised model using reinforcement learning from human feedback, resulting in InstructGPT models.
- Human evaluations show that outputs from the 1.3B parameter InstructGPT model are preferred over those from the 175B GPT-3 despite having fewer parameters.
- InstructGPT models exhibit improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets.
- Fine-tuning with human feedback is a promising direction for aligning language models with human intent on various tasks.

## Summary

The paper titled "Training language models to follow instructions with human feedback" highlights that simply making language models bigger does not necessarily make them better at following a user's intent. Large language models can generate outputs that are untruthful, toxic, or not helpful to the user, indicating misalignment with their intended purpose. The authors propose an approach for aligning language models with user intent on various tasks by fine-tuning them with human feedback. They collect a dataset of labeler demonstrations of desired model behavior and use it to fine-tune GPT-3 using supervised learning. They also collect a dataset of rankings of model outputs and further fine-tune this supervised model using reinforcement learning from human feedback, resulting in InstructGPT models. Human evaluations show that outputs from the 1.3B parameter InstructGPT model are preferred over those from the 175B GPT-3 despite having fewer parameters. Additionally, InstructGPT models exhibit improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. Although InstructGPT still makes simple mistakes, the results indicate that fine-tuning with human feedback is a promising direction for aligning language models with human intent on various tasks.