# Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences

**Authors:** Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, Tengyang Xie  
**arXiv:** [2404.03715](https://arxiv.org/abs/2404.03715)  
**Date:** 2024-04-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.03715v1/

## Key points

- The authors introduce Direct Nash Optimization (DNO) as an algorithm for post-training large language models (LLMs) using preference feedback.
- DNO optimizes over general preferences, leading to monotonic improvement across iterations.
- DNO outperforms GPT-4 and achieves a state-of-the-art win-rate on AlpacaEval 2.0 in experimental trials.
- Observations show that new "large margin" training pairs decrease in quantity as policy improves across iterations in DNO.
- The methodology includes the utilization of an offline dataset for training purposes.

## Summary

, , , , 

In this paper, titled "Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences," the authors explore post-training large language models (LLMs) by utilizing preference feedback from a potent oracle. This facilitates iterative enhancements within the model itself. The conventional method for post-training LLMs involves Reinforcement Learning from Human Feedback (RLHF). However, this approach is limited by "point-wise" rewards and fails to capture intricate preference relationships. To address this, the authors introduce Direct Nash Optimization (DNO), an algorithm that optimizes over general preferences and exhibits monotonic improvement across iterations. In experimental trials, DNO outperforms formidable teachers like GPT-4 and achieves an impressive state-of-the-art win-rate on AlpacaEval 2.0. The paper also discusses observations related to new “large margin” training pairs decreasing in quantity as policy improves across iterations in DNO and highlights the utilization of an offline dataset for training purposes within their methodology.