Attention is not not Explanation
Auteurs : Sarah Wiegreffe, Yuval Pinter
Résumé scientifique
Attention mechanisms play a central role in NLP systems, especially within recurrent neural network (RNN) models. Recently, there has been increasing interest in whether or not the intermediate representations offered by these modules may be used to explain the reasoning for a model's prediction, and consequently reach insights regarding the model's decision-making process. A recent paper claims that `Attention is not Explanation' (Jain and Wallace, 2019). We challenge many of the assumptions underlying this work, arguing that such a claim depends on one's definition of explanation, and that testing it needs to take into account all elements of the model, using a rigorous experimental design. We propose four alternative tests to determine when/whether attention can be used as explanation: a simple uniform-weights baseline; a variance calibration based on multiple random seed runs; a diagnostic framework using frozen weights from pretrained models; and an end-to-end adversarial attention training protocol. Each allows for meaningful interpretation of attention mechanisms in RNN models. We show that even when reliable adversarial distributions can be found, they don't perform well on the simple diagnostic, indicating that prior work does not disprove the usefulness of attention mechanisms for explainability.
Les articles autour de lui, par date et par proximité
Plus haut, c’est plus proche de cet article ; à gauche, c’est plus ancien. Les traits relient les articles proches entre eux : les groupes de traits sont des sous-thèmes. Sélectionnez un point pour voir pourquoi il est là. Retour au résumé de cet article
Les articles autour de celui-ci
Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.
Les articles les plus proches
-
Attention is not ExplanationSarthak Jain, Byron C. Wallace · 2019 · North American Chapter of the Association for Computational Linguistics · cité 2 287 fois · Pas encore résumé
Résumé scientifique
Attention mechanisms have seen wide adoption in neural NLP models. In addition to improving predictive performance, these are often touted as affording transparency: models equipped with attention provide a distribution over attended-to input units, and this is often presented (at least implicitly) as communicating the relative importance of inputs. However, it is unclear what relationship exists between attention weights and model outputs. In this work we perform extensive experiments across a variety of NLP tasks that aim to assess the degree to which attention weights provide meaningful “explanations” for predictions. We find that they largely do not. For example, learned attention weight…
-
Staying True to Your Word: (How) Can Attention Become Explanation?Martin Tutek, J. Šnajder · 2020 · Workshop on Representation Learning for NLP · cité 31 fois · Pas encore résumé
Résumé scientifique
The attention mechanism has quickly become ubiquitous in NLP. In addition to improving performance of models, attention has been widely used as a glimpse into the inner workings of NLP models. The latter aspect has in the recent years become a common topic of discussion, most notably in recent work of Jain and Wallace; Wiegreffe and Pinter. With the shortcomings of using attention weights as a tool of transparency revealed, the attention mechanism has been stuck in a limbo without concrete proof when and whether it can be used as an explanation. In this paper, we provide an explanation as to why attention has seen rightful critique when used with recurrent networks in sequence classification…
-
Is Attention Explanation? An Introduction to the DebateAdrien Bibal, Rémi Cardon, David Alfter et al. · 2022 · Annual Meeting of the Association for Computational Linguistics · cité 182 fois · Pas encore résumé
Résumé scientifique
The performance of deep learning models in NLP and other fields of machine learning has led to a rise in their popularity, and so the need for explanations of these models becomes paramount. Attention has been seen as a solution to increase performance, while providing some explanations. However, a debate has started to cast doubt on the explanatory power of attention in neural networks. Although the debate has created a vast literature thanks to contributions from various areas, the lack of communication is becoming more and more tangible. In this paper, we provide a clear overview of the insights on the debate by critically confronting works from these different areas. This holistic vision…
-
Towards Stable and Explainable Attention MechanismsLijie Hu, Xinhai Wang, Yi-Xin Liu et al. · 2025 · IEEE Transactions on Knowledge and Data Engineering · cité 17 fois · Pas encore résumé
Résumé scientifique
Currently, attention mechanism has become a standard fixture in most state-of-the-art natural language processing (NLP) models, not only due to the outstanding performance it could gain but also due to plausible innate explanations for the behaviors of neural architectures it provides, which is notoriously difficult to analyze. However, recent studies show that attention is unstable against randomness and perturbations during training or testing, such as random seeds and slight perturbation of embedding vectors, which impedes it from becoming a faithful explanation tool. Thus, a natural question is whether we can find some substitute for the current attention that is more stable and could ke…
-
SEAT: Stable and Explainable AttentionLijie Hu, Yi-Xin Liu, Ning-Hao Liu et al. · 2022 · AAAI Conference on Artificial Intelligence · cité 32 fois · Pas encore résumé
Résumé scientifique
Attention mechanism has become a standard fixture in many state-of-the-art natural language processing (NLP) models, not only due to its outstanding performance, but also because it provides plausible innate explanations for neural architectures. However, recent studies show that attention is unstable against randomness and perturbations during training or testing, such as random seeds and slight perturbation of embeddings, which impedes it from being a faithful explanation tool. Thus, a natural question is whether we can find an alternative to vanilla attention, which is more stable and could keep the key characteristics of the explanation. In this paper, we provide a rigorous definition of…
-
Rethinking Attention-Model Explainability through Faithfulness Violation TestY. Liu, Haoliang Li, Yangyang Guo et al. · 2022 · International Conference on Machine Learning · cité 72 fois · Pas encore résumé
Résumé scientifique
Attention mechanisms are dominating the explainability of deep models. They produce probability distributions over the input, which are widely deemed as feature-importance indicators. However, in this paper, we find one critical limitation in attention explanations: weakness in identifying the polarity of feature impact. This would be somehow misleading -- features with higher attention weights may not faithfully contribute to model predictions; instead, they can impose suppression effects. With this finding, we reflect on the explainability of current attention-based techniques, such as Attentio$\odot$Gradient and LRP-based attention explanations. We first propose an actionable diagnostic m…
-
Regularization, Semi-supervision, and Supervision for a Plausible Attention-Based ExplanationDuc Hau Nguyen, Cyrielle Mallart, G. Gravier et al. · 2025 · International Conference on Applications of Natural Language to Data Bases · cité 1 fois · Pas encore résumé
Résumé scientifique
Attention mechanism is contributing to the majority of recent advances in machine learning for natural language processing. Additionally, it results in an attention map that shows the proportional influence of each input in its decision. Empirical studies postulate that attention maps can be provided as an explanation for model output. However, it is still questionable to ask whether this explanation helps regular people to understand and accept the model output (the plausibility of the explanation). Recent studies show that attention weights in the RNN encoders are hardly plausible because they spread on input tokens. We thus propose 3 additional constraints to the learning objective functi…
-
Attention Mechanisms in Deep Learning : Towards Explainable Artificial IntelligenceNour El Houda Dehimi, Zakaria Tolba · 2024 · International Conference on Pattern Analysis and Intelligent Systems · cité 34 fois · Pas encore résumé
Résumé scientifique
Attention mechanisms have revolutionized Machine Learning (ML), particularly in Natural Language Processing (NLP). These mechanisms enable models to selectively focus on crucial parts of the input data, improving performance across tasks like machine translation and sentiment analysis. However, complex ML architectures often remain opaque. This paper explores how attention mechanisms offer a unique path towards Explainable Artificial Intelligence (XAI). By visualizing and analyzing where a model "attends", we can gain insights into which features or data components were most impactful for its predictions. This understanding facilitates model debugging, bias detection, and the development of …
Les travaux antérieurs sur lesquels il s’appuie
-
Reasoning about Entailment with Neural AttentionTim Rocktäschel, Edward Grefenstette, K. Hermann et al. · 2015 · International Conference on Learning Representations · cité 786 fois · Pas encore résumé
Résumé scientifique
While most approaches to automatically recognizing entailment relations have used classifiers employing hand engineered features derived from complex natural language processing pipelines, in practice their performance has been only slightly better than bag-of-word pair classifiers using only lexical similarity. The only attempt so far to build an end-to-end differentiable neural network for entailment failed to outperform such a simple similarity classifier. In this paper, we propose a neural model that reads two sentences to determine entailment using long short-term memory units. We extend this model with a word-by-word neural attention mechanism that encourages reasoning over entailments…
-
Towards A Rigorous Science of Interpretable Machine LearningF. Doshi-Velez, Been Kim · 2017 · arXiv: Machine Learning · preprint, non relu par les pairs · cité 5 838 fois · Pas encore résumé
Résumé scientifique
As machine learning systems become ubiquitous, there has been a surge of interest in interpretable machine learning: systems that provide explanation for their outputs. These explanations are often used to qualitatively assess other criteria such as safety or non-discrimination. However, despite the interest in interpretability, there is very little consensus on what interpretable machine learning is and how it should be measured. In this position paper, we first define interpretability and describe when interpretability is needed (and when it is not). Next, we suggest a taxonomy for rigorous evaluation and expose open questions towards a more rigorous science of interpretable machine learni…
-
Right for the Right Reasons: Training Differentiable Models by Constraining their ExplanationsAndrew Slavin Ross, M. Hughes, F. Doshi-Velez · 2017 · International Joint Conference on Artificial Intelligence · cité 723 fois · Pas encore résumé
Résumé scientifique
Neural networks are among the most accurate supervised learning methods in use today, but their opacity makes them difficult to trust in critical applications, especially when conditions in training differ from those in test. Recent work on explanations for black-box models has produced tools (e.g. LIME) to show the implicit rules behind predictions, which can help us identify when models are right for the wrong reasons. However, these methods do not scale to explaining entire datasets and cannot correct the problems they reveal. We introduce a method for efficiently explaining and regularizing differentiable models by examining and selectively penalizing their input gradients, which provide…
-
Rationalizing Neural PredictionsTao Lei, R. Barzilay, T. Jaakkola · 2016 · Conference on Empirical Methods in Natural Language Processing · cité 922 fois · Pas encore résumé
Résumé scientifique
Prediction without justification has limited applicability. As a remedy, we learn to extract pieces of input text as justifications -- rationales -- that are tailored to be short and coherent, yet sufficient for making the same prediction. Our approach combines two modular components, generator and encoder, which are trained to operate well together. The generator specifies a distribution over text fragments as candidate rationales and these are passed through the encoder for prediction. Rationales are never given during training. Instead, the model is regularized by desiderata for rationales. We evaluate the approach on multi-aspect sentiment analysis against manually annotated test cases. …
-
Generating Token-Level Explanations for Natural Language InferenceJames Thorne, Andreas Vlachos, Christos Christodoulopoulos et al. · 2019 · North American Chapter of the Association for Computational Linguistics · cité 63 fois · Pas encore résumé
Résumé scientifique
The task of Natural Language Inference (NLI) is widely modeled as supervised sentence pair classification. While there has been a lot of work recently on generating explanations of the predictions of classifiers on a single piece of text, there have been no attempts to generate explanations of classifiers operating on pairs of sentences. In this paper, we show that it is possible to generate token-level explanations for NLI without the need for training data explicitly annotated for this purpose. We use a simple LSTM architecture and evaluate both LIME and Anchor explanations for this task. We compare these to a Multiple Instance Learning (MIL) method that uses thresholded attention make tok…
Les articles qui le citent
-
Learning to Explain: Supervised Token Attribution from Transformer Attention PatternsGeorge A. Mihaila · 2026 · arXiv.org · preprint, non relu par les pairs · cité 3 fois · Pas encore résumé
Résumé scientifique
Explainable AI (XAI) has become critical as transformer-based models are deployed in high-stakes applications including healthcare, legal systems, and financial services, where opacity hinders trust and accountability. Transformers self-attention mechanisms have proven valuable for model interpretability, with attention weights successfully used to understand model focus and behavior (Xu et al., 2015); (Wiegreffe and Pinter, 2019). However, existing attention-based explanation methods rely on manually defined aggregation strategies and fixed attribution rules (Abnar and Zuidema, 2020a); (Chefer et al., 2021), while model-agnostic approaches (LIME, SHAP) treat the model as a black box and inc…
-
The Mechanics of Explainability: How Exponential Attention Enables Sparse and Interpretable in-Context LearningYe-Qi Gao, E. Kuruoğlu · 2026 · Image, Video, and Multidimensional Signal Processing Workshop · Pas encore résumé
Résumé scientifique
Post-hoc explainability methods such as LIME and SHAP approximate model behavior externally but cannot reveal the intrinsic decision-making logic of Transformers during incontext learning (ICL). We argue that Softmax attention is not merely a computational component but an inherently interpretable mechanism whose dynamics can be directly read as an explanation. Our key observation is that the exponential nonlinearity in Softmax creates a self-reinforcing feedback loop during optimization: tokens slightly more aligned with the task receive multiplicatively larger gradient updates, causing attention entropy to collapse and producing sparse, human-readable attention maps that transparently indi…
-
Attention in Natural Language ProcessingAndrea Galassi, Marco Lippi, Paolo Torroni · 2020 · IEEE Transactions on Neural Networks and Learning Systems · cité 678 fois · Pas encore résumé
Résumé scientifique
Attention is an increasingly popular mechanism used in a wide range of neural architectures. The mechanism itself has been realized in a variety of formats. However, because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for attention architectures in natural language processing, with a focus on those designed to work with vector representations of the textual data. We propose a taxonomy of attention models according to four dimensions: the representation of the input, the compatibility function, the distribution function, and the multiplicity of the input and/or output. We present the examples of how…
-
A General Survey on Attention Mechanisms in Deep LearningGianni Brauwers, Flavius Frasincar · 2022 · IEEE Transactions on Knowledge and Data Engineering (TKDE), 2021 · cité 720 fois · Résumé IA disponible
Résumé scientifique
Attention is an important mechanism that can be employed for a variety of deep learning models across many different domains and tasks. This survey provides an overview of the most important attention mechanisms proposed in the literature. The various attention mechanisms are explained by means of a framework consisting of a general attention model, uniform notation, and a comprehensive taxonomy of attention mechanisms. Furthermore, the various measures for evaluating attention models are reviewed, and methods to characterize the structure of attention models based on the proposed framework are discussed. Last, future work in the field of attention models is considered.
-
Interpreting Transformers Through Attention Head InterventionMason Kadem, Rong Zheng · 2026 · arXiv.org · preprint, non relu par les pairs · cité 3 fois · Pas encore résumé
Résumé scientifique
Neural networks are growing more capable on their own, but we do not understand their neural mechanisms. Understanding these mechanisms'decision-making processes, or mechanistic interpretability, enables (1) accountability and control in high-stakes domains, (2) the study of digital brains and the emergence of cognition, and (3) discovery of new knowledge when AI systems outperform humans. This paper traces how attention head intervention emerged as a key method for causal interpretability of transformers. The evolution from visualization to intervention represents a paradigm shift from observing correlations to causally validating mechanistic hypotheses through direct intervention. Head int…
Aller plus loin
Comment ces articles sont-ils trouvés ?
Nous comparons le titre et le résumé de cet article avec ceux de tous les articles résumés sur ce site, et avec les articles que Semantic Scholar recommande, ceux qu’il cite et ceux qui le citent (pour un article très cité, ses citations les plus récentes et, depuis OpenAlex, les plus citées). Parmi les travaux antérieurs et les suites, les articles qui s’utilisent en profondeur, une méthode qu’il utilise et les suites très citées remontent un peu. Les niveaux (très proche, proche, apparenté) viennent de la similarité des textes, pas d’un jugement de qualité ; * signale un article sans résumé, comparé par son titre. La phrase sous un article est écrite par une IA à partir des deux résumés : vérifiez-la dans les articles. Un article jugé hors sujet par trois lecteurs quitte la liste.
Mis à jour le 4 novembre 2026.