Rethinking Interpretability in the Era of Large Language Models

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Review Not relevant

   Towards Uncovering How Large Language Model Works: An Explainability Perspective

   Hai-Yan Zhao, Fan Yang, Bo Shen et al. · 2024 · cited 34 times · Not summarized yet

   This explainability review examines how LLM knowledge is encoded and how mechanistic interpretability can uncover their internal workings.

   Abstract

   Large language models (LLMs) have led to breakthroughs in language tasks, yet the internal mechanisms that enable their remarkable generalization and reasoning abilities remain opaque. This lack of transparency presents challenges such as hallucinations, toxicity, and misalignment with human values, hindering the safe and beneficial deployment of LLMs. This paper aims to uncover the mechanisms underlying LLM functionality through the lens of explainability. First, we review how knowledge is architecturally composed within LLMs and encoded in their internal parameters via mechanistic interpretability techniques. Then, we summarize how knowledge is embedded in LLM representations by leveraging…

2. Very close Review Not relevant

   Towards Uncovering How Large Language ModelsWork: An Interpretability Perspective

   Hai-Yan Zhao, Fan Yang, Bo Shen et al. · 2025 · SIGKDD Explorations · cited 2 times · Not summarized yet

   This survey reviews mechanistic interpretability and knowledge encoding to explain opaque LLM behavior.

   Abstract

   Large language models (LLMs) have shown re2 markable performance in tackling natural language 3 tasks, yet the internal mechanisms that enable their 4 impressive generalization and reasoning abilities 5 remain opaque. This lack of transparency presents 6 significant challenges in fundamentally eliminating 7 undesirable behaviors such as hallucinations and 8 toxicity, hindering the safe and beneficial deploy9 ment of LLMs. This survey paper aims to un10 cover the internal working mechanisms underly11 ing LLM functionality through the lens of explain12 ability. First, we review how knowledge is en13 coded within LLMs via mechanistic interpretabil14 ity techniques. Then, we summarize what knowl…

3. Very close Review Not relevant

   Towards Intrinsic Interpretability of Large Language Models:A Survey of Design Principles and Architectures

   Yu-Tong Gao, Qinglin Meng, Yuan Zhou et al. · 2026 · Annual Meeting of the Association for Computational Linguistics · cited 7 times · Not summarized yet

   This survey focuses on intrinsic interpretability built into LLM architectures, rather than the broader uses of LLMs for explanation.

   Abstract

   While Large Language Models (LLMs) have achieved strong performance across many NLP tasks, their opaque internal mechanisms hinder trustworthiness and safe deployment. Existing surveys in explainable AI largely focus on post-hoc explanation methods that interpret trained models through external approximations. In contrast, intrinsic interpretability, which builds transparency directly into model architectures and computations, has recently emerged as a promising alternative. This paper presents a systematic review of the recent advances in intrinsic interpretability for LLMs, categorizing existing approaches into five design paradigms: functional transparency, concept alignment, representati…

4. Very close Review Not relevant

   Local Explanations for Large Language Models: a Brief Review of Methods

   Egor N. Volkov, A. Averkin · 2024 · System Configuration Management · cited 3 times · Not summarized yet

   This brief review classifies methods for producing local explanations of LLM outputs.

   Abstract

   Large Language Models (LLMs) have demonstrated outstanding performance in natural language processing over the past few years. The adoption of LLMs across digital applications in various walks of life is growing exponentially. However, the mechanisms by which LLMs, like all types of artificial neural networks, yield their results remain opaque. This lack of transparency in decision-making poses risks for the further use of the technology. This paper presents a concise review of approaches to obtaining explanations of LLM outcomes. It examines the principal methods of explainable artificial intelligence that are applied to generate explanations for LLM predictions. An approach for classifying…

5. Very close Review Cited by this paper Not relevant

   On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs

   Nitay Calderon, Roi Reichart · 2024 · North American Chapter of the Association for Computational Linguistics · cited 44 times · Not summarized yet

   This paper surveys NLP interpretability through stakeholders’ questions about why, what, and how to explain, and is cited by the paper.

   Abstract

   Recent advancements in NLP systems, particularly with the introduction of LLMs, have led to widespread adoption of these systems by a broad spectrum of users across various domains, impacting decision-making, the job market, society, and scientific research. This surge in usage has led to an explosion in NLP model interpretability and analysis research, accompanied by numerous technical surveys. Yet, these surveys often overlook the needs and perspectives of explanation stakeholders. In this paper, we address three fundamental questions: Why do we need interpretability, what are we interpreting, and how? By exploring these questions, we examine existing interpretability paradigms, their prop…

6. Very close Another approach to the problem Not relevant

   Looking Through the Deep Glasses: How Large Language Models Enhance Explainability of Deep Learning Models

   Philipp Spitzer, S. Celis, Dominik Martin et al. · 2024 · Message Understanding Conference · cited 7 times · Not summarized yet

   This study uses LLM-generated natural-language explanations alongside XAI techniques to help decision-makers understand deep-learning models.

   Abstract

   As AI becomes more powerful, it also becomes more complex. Traditionally, eXplainable AI (XAI) is used to make these models more transparent and interpretable to decision-makers. However, research shows that decision-makers can lack the ability to properly interpret XAI techniques. Large language models (LLMs) offer a solution to this challenge by providing natural language text in combination with XAI techniques to provide more understandable explanations. However, previous work has only explored this approach for inherently interpretable models–an understanding of how LLMs can assist decision-makers when using deep learning models is lacking. To fill this gap, we investigate how different …

7. Very close Review Not relevant

   Towards Transparent AI: A Survey on Explainable Large Language Models

   Avash Palikhe, Zhenyu Yu, Zi-Chong Wang et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 17 times · Not summarized yet

   This survey systematically reviews XAI methods for making LLM decision-making more understandable.

   Abstract

   Large Language Models (LLMs) have played a pivotal role in advancing Artificial Intelligence (AI). However, despite their achievements, LLMs often struggle to explain their decision-making processes, making them a'black box'and presenting a substantial challenge to explainability. This lack of transparency poses a significant obstacle to the adoption of LLMs in high-stakes domain applications, where interpretability is particularly essential. To overcome these limitations, researchers have developed various explainable artificial intelligence (XAI) methods that provide human-interpretable explanations for LLMs. However, a systematic understanding of these methods remains limited. To address …

8. Very close Another approach to the problem Not relevant

   Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention

   Zhen Tan, Tian-Long Chen, Zhenyu (Allen) Zhang et al. · 2023 · AAAI Conference on Artificial Intelligence · cited 33 times · Not summarized yet

   This method uses sparsity-guided interventions to produce holistic explanations of LLMs.

   Abstract

   Large Language Models (LLMs) have achieved unprecedented breakthroughs in various natural language processing domains. However, the enigmatic ``black-box'' nature of LLMs remains a significant challenge for interpretability, hampering transparent and accountable applications. While past approaches, such as attention visualization, pivotal subnetwork extraction, and concept-based analyses, offer some insight, they often focus on either local or global explanations within a single dimension, occasionally falling short in providing comprehensive clarity. In response, we propose a novel methodology anchored in sparsity-guided techniques, aiming to provide a holistic interpretation of LLMs. Our f…

Prior work it builds on

1. Very close Same method, another problem A method it uses Not relevant

   LLMs Understand Glass-Box Models, Discover Surprises, and Suggest Repairs

   Benjamin J. Lengerich, Sebastian Bordt, H. Nori et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 13 times · Not summarized yet

   This work uses LLMs to analyze interpretable glass-box models, identify anomalies, explain them, and suggest repairs; the paper cites it.

   Abstract

   We show that large language models (LLMs) are remarkably good at working with interpretable models that decompose complex outcomes into univariate graph-represented components. By adopting a hierarchical approach to reasoning, LLMs can provide comprehensive model-level summaries without ever requiring the entire model to fit in context. This approach enables LLMs to apply their extensive background knowledge to automate common tasks in data science such as detecting anomalies that contradict prior knowledge, describing potential reasons for the anomalies, and suggesting repairs that would remove the anomalies. We use multiple examples in healthcare to demonstrate the utility of these new cap…

2. Very close Same method, another problem Not relevant

   Augmenting Interpretable Models with LLMs during Training

   Chandan Singh, Armin Askari, Rich Caruana et al. · 2022 · Artificial Intelligence · cited 86 times · Not summarized yet

   This framework uses LLM knowledge to train efficient interpretable models without relying on LLMs at inference, and the paper cites it.

   Abstract

   Recent large language models (LLMs) have demonstrated remarkable prediction performance for a growing array of tasks. However, their proliferation into high-stakes domains (e.g. medicine) and compute-limited settings has created a burgeoning need for interpretability and efficiency. We address this need by proposing Augmented Interpretable Models (Aug-imodels), a framework for leveraging the knowledge learned by LLMs to build extremely efficient and interpretable models. Aug-imodels use LLMs during fitting but not during inference, allowing complete transparency and often a speed/memory improvement of greater than 1,000x for inference compared to LLMs. We explore two instantiations of Aug-im…

3. Close Same method, another problem A method it uses Not relevant

   Post Hoc Explanations of Language Models Can Improve Language Models

   Satyapriya Krishna, Jiaqi W. Ma, Dylan Slack et al. · 2023 · Neural Information Processing Systems · cited 87 times · Not summarized yet

   AMPLIFY generates post-hoc explanations to improve language-model performance, and the paper cites it.

   Abstract

   Large Language Models (LLMs) have demonstrated remarkable capabilities in performing complex tasks. Moreover, recent research has shown that incorporating human-annotated rationales (e.g., Chain-of-Thought prompting) during in-context learning can significantly enhance the performance of these models, particularly on tasks that require reasoning capabilities. However, incorporating such rationales poses challenges in terms of scalability as this requires a high degree of human involvement. In this work, we present a novel framework, Amplifying Model Performance by Leveraging In-Context Learning with Post Hoc Explanations (AMPLIFY), which addresses the aforementioned challenges by automating …

4. Very close Same method, another problem Not relevant

   LLMCheckup: Conversational Examination of Large Language Models via Interpretability Tools and Self-Explanations

   Qianli Wang, Tatiana Anikina, Nils Feldhus et al. · 2024 · HCINLP · cited 24 times · Not summarized yet

   LLMCheckup lets users converse with LLMs about their behavior using interpretability tools and self-explanations, and the paper cites it.

   Abstract

   Interpretability tools that offer explanations in the form of a dialogue have demonstrated their efficacy in enhancing users’ understanding (Slack et al., 2023; Shen et al., 2023), as one-off explanations may fall short in providing sufficient information to the user. Current solutions for dialogue-based explanations, however, often require external tools and modules and are not easily transferable to tasks they were not designed for. With \texttt{LLMCheckup}, we present an easily accessible tool that allows users to chat with any state-of-the-art large language model (LLM) about its behavior. We enable LLMs to generate explanations and perform user intent recognition without fine-tuning, by…

5. Close Groundwork for this paper Not relevant

   Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges

   C. Rudin, Chao-Fan Chen, Zhi Chen et al. · 2021 · Statistics Survey · cited 1,250 times · Not summarized yet

   This paper establishes principles and challenges for interpretable machine learning, which the paper cites as foundational context.

   Abstract

   Interpretability in machine learning (ML) is crucial for high stakes decisions and troubleshooting. In this work, we provide fundamental principles for interpretable ML, and dispel common misunderstandings that dilute the importance of this crucial topic. We also identify 10 technical challenge areas in interpretable machine learning and provide history and background on each problem. Some of these problems are classically important, and some are recent problems that have arisen in the last few years. These problems are: (1) Optimizing sparse logical models such as decision trees; (2) Optimization of scoring systems; (3) Placing constraints into generalized additive models to encourage spars…

Papers that cite it

1. Very close Same method, another problem Not relevant

   Mechanistic interpretability of large language models with applications to the financial services industry

   Ashkan Golgoon, Khashayar Filom, Arjun Ravi Kannan · 2024 · International Conference on AI in Finance · cited 11 times · Not summarized yet

   This study applies mechanistic interpretability to LLMs in financial services, and it cites the paper.

   Abstract

   Large Language Models exhibit remarkable capabilities across a broad spectrum of applications. Nevertheless, due to their intrinsic complexity, these models present substantial challenges in interpreting their internal decision-making processes. This lack of transparency poses critical challenges when it comes to their adaptation by financial institutions, where concerns and accountability regarding bias, fairness, and reliability are of paramount importance. Mechanistic interpretability aims at reverse engineering complex AI models such as transformers. In this paper, we are pioneering the use of mechanistic interpretability to shed some light on the inner workings of large language models …

2. Very close Broader context Not relevant

   Interacting Large Language Model Agents. Interpretable Models and Social Learning

   Adit Jain, Vikram Krishnamurthy · 2024 · arXiv.org · preprint, not peer reviewed · cited 3 times · Not summarized yet

   This paper develops interpretable models for interacting LLM agents and studies bias and herding in social learning; it cites the paper.

   Abstract

   This paper discusses the theory and algorithms for interacting large language model agents (LLMAs) using methods from statistical signal processing and microeconomics. While both fields are mature, their application to decision-making involving interacting LLMAs remains unexplored. Motivated by Bayesian sentiment analysis on online platforms, we construct interpretable models and algorithms that enable LLMAs to interact and perform Bayesian inference. Because interacting LLMAs learn from both prior decisions and external inputs, they can exhibit bias and herding behavior. Thus, developing interpretable models and stochastic control algorithms is essential to understand and mitigate these beh…

3. Close Broader context Uses this paper in depth Not relevant

   Risks, Causes, and Mitigations of Widespread Deployments of Large Language Models (LLMs): A Survey

   Md. Nazmus Sakib, Md Athikul Islam, R. Pathak et al. · 2024 · 2024 2nd International Conference on Artificial Intelligence, Blockchain, and Internet of Things (AIBThings) · cited 16 times · Not summarized yet

   This survey covers broad risks and mitigations of LLM deployment, including reliability and evaluation, and cites the paper.

   Abstract

   Recent advancements in Large Language Models (LLMs), such as ChatGPT and LLaMA, have significantly transformed Natural Language Processing (NLP) with their outstanding abilities in text generation, summarization, and classification. Nevertheless, their widespread adoption introduces numerous challenges, including issues related to academic integrity, copyright, environmental impacts, and ethical considerations such as data bias, fairness, and privacy. The rapid evolution of LLMs also raises concerns regarding the reliability and generalizability of their evaluations. This paper offers a comprehensive survey of the literature on these subjects, systematically gathered and synthesized from Goo…

4. Very close Same method, another problem Not relevant

   Promptexp: Multi-Granularity Prompt Explanation of Large Language Models

   Xi-Ming Dong, Shao-Wei Wang, Da-Yi Lin et al. · 2024 · 2025 2nd IEEE/ACM International Conference on AI-powered Software (AIware) · cited 8 times · Not summarized yet

   PromptExp explains LLM prompts at multiple granularities, addressing the difficulty of interpreting generated token sequences; it cites the paper.

   Abstract

   Large Language Models (LLMs) excel in tasks like natural language understanding and text generation. Prompt engineering plays a critical role in leveraging LLM effectively. However, LLM's black-box nature hinders its interpretability and effective prompt engineering. A wide range of model explanation approaches have been developed for deep learning models (e.g., feature attribution-based and attention-based techniques). However, these local explanations are designed for single-output tasks like classification and regression, and cannot be directly applied to LLMs, which generate sequences of tokens. Recent efforts in LLM explanation focus on natural language explanations, but they are prone …

5. Close Same method, another problem Not relevant

   Beyond Syntax: How Do LLMs Understand Code?

   Marc North, Amir Atapour-Abarghouei, Nelly Bencomo · 2025 · 2025 IEEE/ACM 47th International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · cited 5 times · Not summarized yet

   This study probes LLMs’ internal representations of code concepts and examines attention heads, and it cites the paper.

   Abstract

   Within software engineering research, Large Language Models (LLMs) are often treated as ‘black boxes’, with only their inputs and outputs being considered. In this paper, we take a machine interpretability approach to examine how LLMs internally represent and process code.We focus on variable declaration and function scope, training classifier probes on the residual streams of LLMs as they process code written in different programming languages to explore how LLMs internally represent these concepts across different programming languages. We also look for specific attention heads that support these representations and examine how they behave for inputs of different languages.Our results show…