# IPO: Your Language Model is Secretly a Preference Classifier

**Authors:** Shivank Garg, Ayush Singh, Shweta Singh, Paras Chopra  
**arXiv:** [2502.16182](https://arxiv.org/abs/2502.16182)  
**Date:** 2025-02-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2502.16182v1/

## Key points

- Implicit Preference Optimization (IPO) is a novel approach in reinforcement learning from human feedback (RLHF)
- IPO aims to reduce reliance on external feedback or reward models by using generative LLMs as preference classifiers
- IPO achieved comparable performance to state-of-the-art reward models in obtaining preferences, as shown in a comprehensive evaluation using RewardBench
- IPO outperformed the self-rewarding approach, especially in smaller models, demonstrating robustness and consistency across various tasks and model sizes
- Instruction-based fine-tuning was effective as preference classifiers, with Qwen being a top performer among code-specific models
- Overall, IPO offers a promising alternative approach to RLHF with high performance levels across different tasks and model sizes

## Summary

In the realm of reinforcement learning from human feedback (RLHF), a novel approach known as Implicit Preference Optimization (IPO) has been introduced. This alternative method aims to reduce reliance on external feedback or reward models by leveraging generative LLMs as preference classifiers, ultimately mitigating significant computational and financial costs associated with RLHF. A comprehensive evaluation using RewardBench demonstrated that IPO achieved comparable performance to state-of-the-art reward models for obtaining preferences. Additionally, IPO outperformed the self-rewarding approach, particularly in smaller models, highlighting its robustness and consistency across various tasks and model sizes. Furthermore, instruction-based fine-tuning was found to be effective in acting as preference classifiers, with Qwen emerging as a top performer among code-specific models. These findings showcase how IPO presents a promising alternative approach to RLHF while maintaining high performance levels across different tasks and model sizes.