# Group Robust Preference Optimization in Reward-free RLHF

**Authors:** Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, Ilija Bogunovic  
**arXiv:** [2405.20304](https://arxiv.org/abs/2405.20304)  
**Date:** 2024-05-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.20304v1/

## Key points

- Fine-tuning large language models (LLMs) through reinforcement learning with human feedback (RLHF) on preference data is common practice
- Preference data often come from diverse groups of labelers, encompassing various demographics, ethnicities, and company teams
- Traditional RLHF approaches typically use a "one-size-fits-all" strategy without considering unique characteristics and needs of different groups
- Group Robust Preference Optimization (GRPO) method aims to align LLMs with preferences of individual groups in a robust manner
- GRPO seeks a robust policy that maximizes worst-case group performance by adaptively weighting importance of different groups and prioritizing those with worse cumulative loss
- Theoretical studies show GRPO's convergence within the log-linear policy class
- Fine-tuning LLMs using diverse group-based global opinion data with GRPO leads to significant improvements in performance for worst-performing groups, reduced loss imbalances across groups, and enhanced probability accuracies compared to non-robust baselines
- Authors believe GRPO holds promise for tailored LLM fine-tuning endeavors to meet specific needs of diverse teams and user groups
- In broader context, GRPO shows potential for mitigating biases and enhancing alignment performance in applications involving large language models

## Summary

In the realm of adapting large language models (LLMs) for specific tasks, fine-tuning through reinforcement learning with human feedback (RLHF) on preference data is a common practice. However, these preference data often stem from diverse groups of labelers, encompassing various demographics, ethnicities, and company teams. Traditional RLHF approaches typically adopt a "one-size-fits-all" strategy, assuming and optimizing a single preference model without considering the unique characteristics and needs of different groups. To address this limitation, a novel Group Robust Preference Optimization (GRPO) method has been proposed to effectively align LLMs with the preferences of individual groups in a robust manner. The GRPO approach builds upon reward-free direct preference optimization methods but distinguishes itself by seeking a robust policy that maximizes the worst-case group performance. By adaptively and sequentially weighting the importance of different groups and prioritizing those with worse cumulative loss, GRPO aims to achieve alignment while minimizing disparities across groups. The feasibility of GRPO has been theoretically studied, particularly focusing on its convergence within the log-linear policy class. Through fine-tuning LLMs using diverse group-based global opinion data with GRPO, significant improvements have been observed in performance for the worst-performing groups. Additionally, there has been a reduction in loss imbalances across groups and enhanced probability accuracies compared to non-robust baselines. The authors believe that this approach holds promise for future tailored LLM fine-tuning endeavors aimed at meeting the specific needs of diverse teams and user groups. In a broader context, GRPO shows potential for mitigating biases and enhancing alignment performance in various applications involving large language models.