# Learning to Deceive with Attention-Based Explanations

**Authors:** Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, Zachary C. Lipton  
**arXiv:** [1909.07913](https://arxiv.org/abs/1909.07913)  
**Date:** 2019-09-17  
**Summary page:** https://www.summarizepaper.com/en/paper/1909.07913v2/

## Key points

- The paper explores the use of attention mechanisms in neural architectures for natural language processing
- The authors question the reliability of attention weights in explaining model decisions
- They propose a method for training models to generate deceptive attention masks
- Manipulating attention weights has minimal impact on accuracy across multiple models and tasks
- Human study shows that manipulated attention-based explanations deceive people into thinking biased predictions do not rely on gender
- These findings cast doubt on the reliability of attention as a tool for auditing algorithms in terms of fairness and accountability
- The research highlights potential limitations in using attention mechanisms for interpretability purposes
- Raises important questions about algorithmic transparency

## Summary

The paper titled "Learning to Deceive with Attention-Based Explanations" explores the use of attention mechanisms in neural architectures for natural language processing. While attention weights are commonly used to improve predictive accuracy and provide interpretability, the authors question their reliability in explaining model decisions. They propose a method for training models to generate deceptive attention masks, reducing the weight assigned to certain impermissible tokens while still relying on them for predictions. Surprisingly, this manipulation of attention weights has minimal impact on accuracy across multiple models and tasks. Through a human study, the authors demonstrate that these manipulated attention-based explanations deceive people into thinking that predictions from a biased model do not rely on gender. These findings cast doubt on the reliability of attention as a tool for auditing algorithms in terms of fairness and accountability. Overall, this research highlights potential limitations in using attention mechanisms for interpretability purposes and raises important questions about algorithmic transparency.