# Scaling Laws for Reward Model Overoptimization

**Authors:** Leo Gao, John Schulman, Jacob Hilton  
**arXiv:** [2210.10760](https://arxiv.org/abs/2210.10760)  
**Date:** 2022-10-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2210.10760v1/

## Key points

- Optimizing against a reward model trained to predict human preferences can lead to overoptimization and hinder performance of the ground truth
- The study uses a synthetic setup with a fixed "gold-standard" reward model as a substitute for humans
- Two different methods, reinforcement learning and best-of-$n$ sampling, are used to optimize against the proxy reward model
- The relationship between gold reward model scores and proxy reward model scores depends on the optimization method used
- Coefficients in this relationship scale smoothly with the number of reward model parameters
- Factors such as size of the reward model dataset, number of reward model and policy parameters, and coefficient of KL penalty impact this relationship
- Implications for theoretical considerations in AI alignment are discussed based on empirical results
- Optimizing against imperfect proxy models can affect performance in reinforcement learning from human feedback scenarios
- Balancing optimization against such models while maintaining alignment with ground truth objectives is important.

## Summary

In reinforcement learning from human feedback, optimizing against a reward model trained to predict human preferences can lead to overoptimization and hinder the performance of the ground truth. To address this challenge, the authors of this study use a synthetic setup where a fixed "gold-standard" reward model acts as a substitute for humans and provides labels for training a proxy reward model. The study investigates how the gold reward model score changes when optimizing against the proxy reward model using two different methods: reinforcement learning and best-of-$n$ sampling. The authors find that the relationship between these scores follows a different functional form depending on the optimization method used. Additionally, they observe that the coefficients in this relationship scale smoothly with the number of reward model parameters. Various factors such as size of the reward model dataset, number of reward model and policy parameters, and coefficient of KL penalty added to the reward in reinforcement learning are explored to understand their impact on this relationship. By analyzing these empirical results, implications for theoretical considerations in AI alignment are discussed. Overall, this study provides insights into how optimizing against imperfect proxy models can affect performance in reinforcement learning from human feedback scenarios. The findings highlight important considerations for balancing optimization against such models while maintaining alignment with ground truth objectives.