# On the Exploitability of Instruction Tuning

**Authors:** Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, Tom Goldstein  
**arXiv:** [2306.17194](https://arxiv.org/abs/2306.17194)  
**Date:** 2023-06-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.17194v1/

## Key points

- The paper explores the concept of instruction tuning in language models.
- Adversaries can exploit instruction tuning by injecting specific examples into the training data to change the model's behavior.
- Content injection is one way adversaries achieve this, by injecting training examples that mention target content and elicit desired behavior from downstream models.
- The authors propose an automated data poisoning pipeline called AutoPoison to address this issue.
- AutoPoison incorporates versatile attack goals into poisoned data using an oracle LLM, allowing adversaries to achieve their desired exploitable behavior.
- Two example attacks showcased are content injection and over-refusal attacks, aiming to induce specific exploitable behaviors in the model.
- AutoPoison enables adversaries to change a model's behavior by poisoning only a small fraction of data while maintaining stealthiness in the poisoned examples.
- Data quality plays a crucial role in the behavior of instruction-tuned models and responsible deployments of LLMs.
- The research sheds light on the exploitability of instruction tuning and provides insights into how adversaries can manipulate LLMs through injected examples.
- The proposed AutoPoison pipeline offers a powerful tool for achieving these manipulations while maintaining stealthiness.

## Summary

The paper titled "On the Exploitability of Instruction Tuning" explores the concept of instruction tuning, a technique used to align large language models (LLMs) with human intents. The authors investigate how an adversary can exploit this technique by injecting specific instruction-following examples into the training data in order to intentionally change the behavior of the model. One way that adversaries can achieve this is through content injection, where they inject training examples that mention target content and elicit a desired behavior from downstream models. To address this issue, the authors propose an automated data poisoning pipeline called AutoPoison. This pipeline incorporates versatile attack goals into poisoned data using an oracle LLM, allowing adversaries to achieve their desired exploitable behavior. The paper showcases two example attacks: content injection and over-refusal attacks. Each attack aims to induce a specific exploitable behavior in the model. The authors quantify and benchmark the strength and stealthiness of their data poisoning scheme. The results demonstrate that AutoPoison enables adversaries to change a model's behavior by poisoning only a small fraction of data while maintaining a high level of stealthiness in the poisoned examples. This highlights how data quality affects the behavior of instruction-tuned models and emphasizes the importance of data quality for responsible deployments of LLMs. In conclusion, this work sheds light on the exploitability of instruction tuning and provides insights into how adversaries can manipulate LLMs by injecting specific examples into training data. The proposed AutoPoison pipeline offers a powerful tool for achieving these manipulations while maintaining stealthiness. By raising awareness about these vulnerabilities, this research contributes to responsible deployments of LLMs and provides code for AutoPoison at [https://github.com/azshue/AutoPoison](https://github.com/azshue/AutoPoison).