# Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning

**Authors:** Ruiqi Zhang, Licong Lin, Yu Bai, Song Mei  
**arXiv:** [2404.05868](https://arxiv.org/abs/2404.05868)  
**Date:** 2024-04-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.05868v1/

## Key points

- Authors address the issue of Large Language Models (LLMs) memorizing sensitive, private, or copyrighted data during pre-training
- Focus of LLM unlearning is to remove undesirable data while maintaining utility for other tasks
- Proposed Negative Preference Optimization (NPO) method aims to efficiently and effectively unlearn specific datasets
- Theoretical analysis shows that minimizing NPO loss leads to slower progression towards catastrophic collapse compared to gradient ascent (GA)
- Experimental results demonstrate that NPO-based approaches strike a better balance between eliminating undesirable data and preserving model utilities
- NPO-based methods produce more coherent outputs and achieve notable success in forgetting 50% or more of training data on the TOFU dataset

## Summary

In their paper titled "Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning," authors Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei address the issue of Large Language Models (LLMs) memorizing sensitive, private, or copyrighted data during pre-training. The focus of LLM unlearning is to remove the influence of undesirable data from these models while maintaining their utility for other tasks. Previous methods for LLM unlearning have primarily relied on gradient ascent (GA) on the loss of undesirable data. However, they often struggle with effectively unlearning target data or experience catastrophic collapse. To tackle these challenges, the authors propose Negative Preference Optimization (NPO), a straightforward alignment-inspired method designed to efficiently and effectively unlearn a specific dataset. Through theoretical analysis, they demonstrate that minimizing the NPO loss leads to a significantly slower progression towards catastrophic collapse compared to GA. Experimental results on synthetic data and the TOFU dataset showcase that NPO-based approaches strike a better balance between eliminating undesirable data and preserving model utilities. Moreover, the authors observe that NPO-based methods produce more coherent outputs compared to GA-based techniques, which often generate nonsensical results. Notably, on the TOFU dataset, NPO-based methods achieve notable success in forgetting 50% or more of the training data – a significant improvement over existing methods struggling with forgetting just 10% of training data. Overall,this study introduces an innovative approach that enhances the effectiveness and efficiency of unlearning undesirable data from LLMs while maintaining their performance on various tasks.