Inspecting and Editing Knowledge Representations in Language Models

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Close Not relevant

   AnyEdit: Edit Any Knowledge Encoded in Language Models

   Houcheng Jiang, Junfeng Fang, Ningyu Zhang et al. · 2025 · Computation and Language · Not summarized yet

   Abstract

   Large language models (LLMs) often produce incorrect or outdated information, necessitating efficient and precise knowledge updates. Current model editing methods, however, struggle with long-form knowledge in diverse formats, such as poetry, code snippets, and mathematical derivations. These limitations arise from their reliance on editing a single token's hidden state, a limitation we term "efficacy barrier". To solve this, we propose AnyEdit, a new autoregressive editing paradigm. It decomposes long-form knowledge into sequential chunks and iteratively edits the key token in each chunk, ensuring consistent and accurate outputs. Theoretically, we ground AnyEdit in the Chain Rule of Mutual …

2. Close Not relevant

   LMExplainer: a Knowledge-Enhanced Explainer for Language Models

   Zichen Chen, Ambuj K Singh, Misha Sra · 2023 · Computation and Language · AI summary available

   Abstract

   Large language models (LMs) such as GPT-4 are very powerful and can process different kinds of natural language processing (NLP) tasks. However, it can be difficult to interpret the results due to the multi-layer nonlinear model structure and millions of parameters. Lack of understanding of how the model works can make the model unreliable and dangerous for everyday users in real-world scenarios. Most recent works exploit the weights of attention to provide explanations for model predictions. However, pure attention-based explanation is unable to support the growing complexity of the models, and cannot reason about their decision-making processes. Thus, we propose LMExplainer, a knowledge-en…

3. Close Not relevant

   Fine-tuning Language Models for Factuality

   Katherine Tian, Eric Mitchell, Huaxiu Yao et al. · 2023 · Computation and Language · AI summary available

   Abstract

   The fluency and creativity of large pre-trained language models (LLMs) have led to their widespread use, sometimes even as a replacement for traditional search engines. Yet language models are prone to making convincing but factually inaccurate claims, often referred to as 'hallucinations.' These errors can inadvertently spread misinformation or harmfully perpetuate misconceptions. Further, manual fact-checking of model responses is a time-consuming process, making human factuality labels expensive to acquire. In this work, we fine-tune language models to be more factual, without human labeling and targeting more open-ended generation settings than past work. We leverage two key recent innov…

4. Close Not relevant

   Language Models as Knowledge Bases?

   Fabio Petroni, Tim Rocktäschel, Patrick Lewis et al. · 2019 · Computation and Language · AI summary available

   Abstract

   Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as "fill-in-the-blank" cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of relations, are easy to extend to more data, and require no human supervision to train. We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of s…

5. Close Not relevant

   Monitoring Latent World States in Language Models with Propositional Probes

   Jiahai Feng, Stuart Russell, Jacob Steinhardt · 2024 · Computation and Language · AI summary available

   Abstract

   Language models are susceptible to bias, sycophancy, backdoors, and other tendencies that lead to unfaithful responses to the input context. Interpreting internal states of language models could help monitor and correct unfaithful behavior. We hypothesize that language models represent their input contexts in a latent world model, and seek to extract this latent world state from the activations. We do so with 'propositional probes', which compositionally probe tokens for lexical information and bind them into logical propositions representing the world state. For example, given the input context ''Greg is a nurse. Laura is a physicist.'', we decode the propositions ''WorksAs(Greg, nurse)'' a…

6. Close Not relevant

   Knowledge Infused Decoding

   Ruibo Liu, Guoqing Zheng, Shashank Gupta et al. · 2022 · Computation and Language · AI summary available

   Abstract

   Pre-trained language models (LMs) have been shown to memorize a substantial amount of knowledge from the pre-training corpora; however, they are still limited in recalling factually correct knowledge given a certain context. Hence, they tend to suffer from counterfactual or hallucinatory generation when used in knowledge-intensive natural language generation (NLG) tasks. Recent remedies to this problem focus on modifying either the pre-training or task fine-tuning objectives to incorporate knowledge, which normally require additional costly training or architecture modification of LMs for practical applications. We present Knowledge Infused Decoding (KID) -- a novel decoding algorithm for ge…

7. Close Not relevant

   Locating and Editing Factual Knowledge in GPT

   Kevin Meng, David Bau, Alex Andonian et al. · 2022 · Computation and Language · AI summary available

   Abstract

   We investigate the mechanisms underlying factual knowledge recall in autoregressive transformer language models. First, we develop a causal intervention for identifying neuron activations capable of altering a model's factual predictions. Within large GPT-style models, this reveals two distinct sets of neurons that we hypothesize correspond to knowing an abstract fact and saying a concrete word, respectively. This insight inspires the development of ROME, a novel method for editing facts stored in model weights. For evaluation, we assemble CounterFact, a dataset of over twenty thousand counterfactuals and tools to facilitate sensitive measurements of knowledge editing. Using CounterFact, we …

8. Close Not relevant

   Generated Knowledge Prompting for Commonsense Reasoning

   Jiacheng Liu, Alisa Liu, Ximing Lu et al. · 2021 · Computation and Language · AI summary available

   Abstract

   Despite their ability to capture large amount of knowledge during pretraining, large-scale language models often benefit from incorporating external knowledge bases, especially on commonsense reasoning tasks. This motivates us to explore how we can best leverage knowledge elicited from language models themselves. We propose generating knowledge statements directly from a language model with a generic prompt format, then selecting the knowledge which maximizes prediction probability. Despite its simplicity, this approach improves performance of both off-the-shelf and finetuned language models on four commonsense reasoning tasks, improving the state-of-the-art on numerical commonsense (NumerSe…