# What Makes and Breaks Safety Fine-tuning? A Mechanistic Study

**Authors:** Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip H. S. Torr, Amartya Sanyal, Puneet K. Dokania  
**arXiv:** [2407.10264](https://arxiv.org/abs/2407.10264)  
**Date:** 2024-07-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.10264v3/

## Key points

- A study on safety fine-tuning methods for Large Language Models (LLMs) was conducted by Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip H. S. Torr, Amartya Sanyal, and Puneet K. Dokania.
- The researchers developed a synthetic data generation framework to understand factors contributing to model safety through safety fine-tuning.
- The investigation explored three prominent safety fine-tuning techniques: supervised safety fine-tuning, direct preference optimization, and unlearning.
- These methods induce minimal transformations in Multi-Layer Perceptron (MLP) weights to align unsafe inputs within the null space of the weights.
- The model processes potentially harmful inputs as if they were safe based on this clustering mechanism.
- The researchers applied their methodology on real-world models Llama-2 7B and Llama-3 8B to validate their findings.

## Summary

A study on safety fine-tuning methods for Large Language Models (LLMs) was conducted by Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip H. S. Torr, Amartya Sanyal, and Puneet K. Dokania to ensure their alignment with human preferences for safe deployment. The researchers developed a synthetic data generation framework to understand the factors that contribute to model safety through safety fine-tuning. This framework aims to capture crucial aspects of unsafe inputs by examining how models interact with different tasks and concepts they are required to process. The investigation involved exploring three prominent safety fine-tuning techniques: supervised safety fine-tuning, direct preference optimization, and unlearning. The researchers found that these methods induce minimal transformations in Multi-Layer Perceptron (MLP) weights to specifically align unsafe inputs within the null space of the weights. This results in a clustering of inputs based on whether the model categorizes them as safe or unsafe. Furthermore, when presented with adversarial inputs such as a jailbreak scenario, the model's activations tend to be closer to safer samples due to this alignment within the null space of MLP weights. As a result, the model processes potentially harmful inputs as if they were safe based on this clustering mechanism. To validate their findings, the researchers applied their methodology on real-world models Llama-2 7B and Llama-3 8B. Through this comprehensive study, they shed light on how safety fine-tuning mechanisms operate and influence model behavior towards ensuring safer deployment practices for LLMs in various applications.