# Prompt Leakage effect and defense strategies for multi-turn LLM interactions

**Authors:** Divyansh Agarwal, Alexander R. Fabbri, Ben Risher, Philippe Laban, Shafiq Joty, Chien-Sheng Wu  
**arXiv:** [2404.16251](https://arxiv.org/abs/2404.16251)  
**Date:** 2024-04-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.16251v3/

## Key points

- Prompt leakage in large language models (LLMs) is a significant security and privacy concern, potentially compromising sensitive intellectual property and aiding malicious attackers.
- There is a lack of systematic evaluation of prompt leakage threats and effective mitigation strategies in multi-turn interactions using LLMs.
- The study titled "Prompt Leakage effect and defense strategies for multi-turn LLM interactions" by Divyansh Agarwal et al. explores vulnerabilities associated with prompt leakage across 10 closed- and open-source LLMs in four different domains.
- A novel threat model leveraging the sycophancy effect within LLMs significantly increases the average attack success rate (ASR) from 17.7% to 86.2% in multi-turn scenarios.
- The researchers dissect specific prompt contents like task instructions and knowledge documents to understand the extent of leakage, evaluate seven black-box defense strategies, and fine-tune an open-source model to enhance resilience against prompt leakage attempts.
- Various combinations of defenses against their threat model are presented along with a cost analysis to guide the development of secure LLM applications.
- The research provides valuable insights for enhancing security in multi-turn LLM interactions, offering directions for future studies in this field.

## Summary

, , , , 

Prompt leakage in large language models (LLMs) presents a significant security and privacy concern, as it can potentially compromise sensitive intellectual property and serve as reconnaissance for malicious attackers. Despite the growing use of LLMs in various applications, there is a lack of systematic evaluation of prompt leakage threats and effective mitigation strategies, particularly in multi-turn interactions. In their study titled "Prompt Leakage effect and defense strategies for multi-turn LLM interactions," authors Divyansh Agarwal, Alexander R. Fabbri, Ben Risher, Philippe Laban, Shafiq Joty, and Chien-Sheng Wu delve into the vulnerabilities associated with prompt leakage across 10 closed- and open-source LLMs spanning four different domains. They introduce a novel threat model that capitalizes on the sycophancy effect within LLMs to significantly increase the average attack success rate (ASR) from 17.7% to 86.2% in multi-turn scenarios. Through a meticulously designed experimental setup, the researchers are able to dissect specific prompt contents such as task instructions and knowledge documents to better understand the extent of leakage. They evaluate the effectiveness of seven black-box defense strategies and explore the process of fine-tuning an open-source model to enhance its resilience against prompt leakage attempts. Furthermore, the study presents various combinations of defenses against their threat model along with a cost analysis to provide insights into building secure LLM applications. By shedding light on key takeaways for enhancing security in multi-turn LLM interactions, this research contributes valuable directions for future studies in this domain.