# Adversarial Fine-Tuning of Language Models: An Iterative Optimisation Approach for the Generation and Detection of Problematic Content

**Authors:** Charles O'Neill, Jack Miller, Ioana Ciuca, Yuan-Sen Ting, Thang Bui  
**arXiv:** [2308.13768](https://arxiv.org/abs/2308.13768)  
**Date:** 2023-08-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.13768v1/

## Key points

- Authors address the issue of unintended harmful content produced by Large Language Models (LLMs)
- Propose a unique dual-stage optimisation technique using adversarial fine-tuning
- Approach involves an adversarial model generating potentially harmful prompts and a judge model identifying these prompts iteratively
- Adversarial cycle creates a diverse dataset for further fine-tuning, allowing for continuous improvement
- Method evaluated through classification accuracy on dataset containing problematic prompts undetected by GPT-4 and harmless prompts
- Significant increase in judge model's classification accuracy observed during optimisation
- Basic model \texttt{ada} achieves higher accuracy than GPT-4 on hold-out test set after refinement rounds
- Fine-tuning process enhances performance in related tasks like identifying toxic comments
- Dual-stage optimisation technique using adversarial fine-tuning proves effective in addressing unintended harmful content generation in LLMs and improves performance in parallel tasks

## Summary

In their paper "Adversarial Fine-Tuning of Language Models: An Iterative Optimisation Approach for the Generation and Detection of Problematic Content," authors Charles O'Neill, Jack Miller, Ioana Ciuca, Yuan-Sen Ting, and Thang Bui from various departments at the Australian National University address the issue of unintended harmful content produced by Large Language Models (LLMs). They propose a unique dual-stage optimisation technique using adversarial fine-tuning to tackle this challenge. Their approach involves an adversarial model trained to generate potentially harmful prompts and a judge model iteratively refined to accurately identify these prompts. Through an adversarial cycle where these models compete in prompt generation, a diverse dataset is created for further fine-tuning. This process allows for continuous improvement and enhanced performance. The effectiveness of their method is evaluated through classification accuracy on a dataset containing problematic prompts undetected by GPT-4 as well as contentious yet harmless prompts. The results demonstrate a significant increase in classification accuracy of the judge model as it undergoes optimisation. Additionally, they show that even a basic model named \texttt{ada} can achieve higher accuracy on a hold-out test set compared to GPT-4 after just a few rounds of refinement. Furthermore, this fine-tuning process enhances performance in related tasks such as identifying toxic comments. Overall, the authors' dual-stage optimisation technique using adversarial fine-tuning proves effective in addressing the challenge of unintended harmful content generation in LLMs and shows potential for enhancing performance in parallel tasks.