# Transforming and Combining Rewards for Aligning Large Language Models

**Authors:** Zihao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alex D'Amour, Sanmi Koyejo, Victor Veitch  
**arXiv:** [2402.00742](https://arxiv.org/abs/2402.00742)  
**Date:** 2024-02-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.00742v1/

## Key points

- Study focuses on aligning language models with human preferences
- Authors learn a reward model from preference data and update the language model
- Address two key issues: monotone transformations of reward model and combining multiple reward models
- Probabilistic interpretation used for alignment procedure
- Transformation emphasizes improving poorly-performing outputs to address underfitting and reward hacking
- Aggregation of rewards linked to logical conjunction for meeting all measured properties
- Significant improvements observed in aligning language models to be helpful and harmless using RLHF
- Refined approach enhances performance and mitigates potential pitfalls like underfitting and reward hacking
- Related work highlights research on mitigating reward hacking in the RLHF pipeline
- Techniques explored include reward model averaging, constrained optimization, reward model regularization, iterative human preference collection, and data bias mitigation

## Summary

In the study "Transforming and Combining Rewards for Aligning Large Language Models," the authors delve into the process of aligning language models with human preferences. They first learn a reward model from preference data and then use this model to update the language model. The authors address two key issues that arise in this approach: monotone transformations of the reward model and combining multiple reward models when aligning language models to multiple properties. By employing a probabilistic interpretation of the alignment procedure, they identify a natural choice for transforming rewards learned from Bradley-Terry preference models. This transformation emphasizes improving poorly-performing outputs rather than those already scoring well, addressing underfitting and reward hacking issues. Additionally, it enables principled aggregation of rewards by linking summation to logical conjunction, ensuring that the sum of transformed rewards corresponds to the probability that the output meets all measured properties. Through experiments using RLHF (Reward Learning from Human Feedback) to align language models to be both helpful and harmless, significant improvements over baseline approaches are observed. The refined approach not only enhances performance but also mitigates potential pitfalls such as underfitting and reward hacking. Furthermore, in discussing related work, the authors highlight a growing body of research on mitigating reward hacking in the RLHF pipeline. Techniques such as reward model averaging, constrained optimization, reward model regularization, iterative human preference collection, and data bias mitigation have been explored in efforts to enhance alignment between language models and human preferences. These techniques contribute valuable insights towards refining alignment processes and addressing challenges associated with optimizing language models based on human feedback.