AI summary · arXiv: 1909.01066v1 · cs.CL
Language Models as Knowledge Bases?
The essentials
- Recent progress in pretraining language models on large textual corpora has resulted in significant advancements in downstream NLP tasks
- Language models learn linguistic knowledge and store relational knowledge from the training data
- Language models offer advantages over structured knowledge bases, such as no requirement for schema engineering, ability to query about an open class of relations, easy scalability to more data, and no need for human supervision during training
- BERT exhibits competitive relational knowledge compared to traditional NLP methods even without fine-tuning
- BERT performs remarkably well on open-domain question answering compared to a supervised baseline
- Certain types of factual knowledge are learned more readily than others through standard language model pretraining approaches
- Pretrained language models have potential as unsupervised open-domain question answering systems
- The researchers provide code for reproducing their analysis at https://github.com/facebookresearch/LAMA
Key points written by AI: they may contain errors, check them against the paper.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Among the 31 papers around this one, the name that comes back most: Adam Roberts (2 papers)
3 more people sign several of them. The full map adds their institutions and approaches, from the papers of the last years.
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
How Much Knowledge Can You Pack into the Parameters of a Language Model?Adam Roberts, Colin A. Raffel, Noam Shazeer · 2020 · Conference on Empirical Methods in Natural Language Processing · cited 1,157 times · Not summarized yet
How Much Knowledge Can You Pack into the Parameters of a Language Model? extends parametric knowledge probing by fine-tuning models for closed-book open-domain question answering.
Abstract
It has recently been observed that neural language models trained on unstructured text can implicitly store and retrieve knowledge using natural language queries. In this short paper, we measure the practical utility of this approach by fine-tuning pre-trained models to answer questions without access to any external context or knowledge. We show that this approach scales surprisingly well with model size and outperforms models that explicitly look up knowledge on the open-domain variants of Natural Questions and WebQuestions. To facilitate reproducibility and future work, we release our code and trained models.
-
REALM: Retrieval-Augmented Language Model Pre-TrainingKelvin Guu, Kenton Lee, Zora Tung et al. · 2020 · International Conference on Machine Learning · cited 3,608 times · Not summarized yet
REALM augments language-model pretraining with a retriever over documents, addressing factual knowledge access through retrieval rather than relying only on model parameters.
Abstract
Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language modeling a…
-
How to Query Language Models?Leonard Adolphs, S. Dhuliawala, Thomas Hofmann · 2021 · arXiv.org · preprint, not peer reviewed · cited 17 times · Not summarized yet
How to Query Language Models? uses example-based prompts to improve access to factual knowledge through cloze-style queries.
Abstract
Large pre-trained language models (LMs) are capable of not only recovering linguistic but also factual and commonsense knowledge. To access the knowledge stored in mask-based LMs, we can use cloze-style questions and let the model fill in the blank. The flexibility advantage over structured knowledge bases comes with the drawback of finding the right query for a certain information need. Inspired by human behavior to disambiguate a question, we propose to query LMs by example. To clarify the ambivalent question"Who does Neuer play for?", a successful strategy is to demonstrate the relation using another subject, e.g.,"Ronaldo plays for Portugal. Who does Neuer play for?". We apply this appro…
-
Language Models are Open Knowledge GraphsChenguang Wang, Xiao Liu, D. Song · 2020 · arXiv.org · preprint, not peer reviewed · cited 152 times · Not summarized yet
Language Models are Open Knowledge Graphs extracts relational knowledge from pretrained language models to construct knowledge graphs without human supervision.
Abstract
This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e.g., BERT, GPT-2/3), without human supervision. Popular KGs (e.g, Wikidata, NELL) are built in either a supervised or semi-supervised manner, requiring humans to create knowledge. Recent deep language models automatically acquire knowledge from large-scale corpora via pre-training. The stored knowledge has enabled the language models to improve downstream NLP tasks, e.g., answering questions, and writing code and articles. In this paper, we propose an unsupervised method to cast the knowledge contained within language models into KGs. We show that KGs are constructed with a single forward pass of the …
-
Structured Knowledge Grounding for Question AnsweringYu-Jie Lu, Siqi Ouyang, Kai-Rui Zhou · 2022 · cited 3 times · Not summarized yet
Structured Knowledge Grounding for Question Answering asks how language models can use relational knowledge for knowledge-based question answering.
Abstract
Can language models (LM) ground question-answering (QA) tasks in the knowledge base via inherent relational reasoning ability? While previous models that use only LMs have seen some success on many QA tasks, more recent methods include knowledge graphs (KG) to complement LMs with their more logic-driven implicit knowledge. However, effectively extracting information from structured data, like KGs, empowers LMs to remain an open question, and current models rely on graph techniques to extract knowledge. In this paper, we propose to solely leverage the LMs to combine the language and knowledge for knowledge based question-answering with flexibility, breadth of coverage and structured reasoning…
-
Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval AugmentationRuiyang Ren, Yuhao Wang, Yingqi Qu et al. · 2023 · Computation and Language · cited 214 times · AI summary available
Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation studies factual knowledge in language models and its effect on open-domain QA.
Abstract
Knowledge-intensive tasks (e.g., open-domain question answering (QA)) require a substantial amount of factual knowledge and often rely on external information for assistance. Recently, large language models (LLMs) (e.g., ChatGPT), have demonstrated impressive prowess in solving a wide range of tasks with world knowledge, including knowledge-intensive tasks. However, it remains unclear how well LLMs are able to perceive their factual knowledge boundaries, particularly how they behave when incorporating retrieval augmentation. In this study, we present an initial analysis of the factual knowledge boundaries of LLMs and how retrieval augmentation affects LLMs on open-domain QA. Specially, we fo…
-
Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA?Cunxiang Wang, Pai Liu, Yue Zhang · 2021 · Annual Meeting of the Association for Computational Linguistics · cited 97 times · Not summarized yet
Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA? evaluates whether pretrained models retain facts needed to answer questions without external context.
Abstract
Recent work has investigated the interesting question using pre-trained language models (PLMs) as knowledge bases for answering open questions. However, existing work is limited in using small benchmarks with high test-train overlaps. We construct a new dataset of closed-book QA using SQuAD, and investigate the performance of BART. Experiments show that it is challenging for BART to remember training facts in high precision, and also challenging to answer closed-book questions even if relevant knowledge is retained. Some promising directions are found, including decoupling the knowledge memorizing process and the QA finetune process, forcing the model to recall relevant knowledge when questi…
-
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric MemoriesAlex Troy Mallen, Akari Asai, Victor Zhong et al. · 2022 · Annual Meeting of the Association for Computational Linguistics · cited 1,634 times · Not summarized yet
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories probes factual recall and compares it with retrieval augmentation on entity-centric QA.
Abstract
Despite their impressive performance on diverse tasks, large language models (LMs) still struggle with tasks requiring rich world knowledge, implying the difficulty of encoding a wealth of world knowledge in their parameters. This paper aims to understand LMs’ strengths and limitations in memorizing factual knowledge, by conducting large-scale knowledge probing experiments on two open-domain entity-centric QA datasets: PopQA, our new dataset with 14k questions about long-tail entities, and EntityQuestions, a widely used open-domain QA dataset. We find that LMs struggle with less popular factual knowledge, and that retrieval augmentation helps significantly in these cases. Scaling, on the oth…
Prior work it builds on
-
BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingJ. Devlin, Ming-Wei Chang, Kenton Lee et al. · 2019 · North American Chapter of the Association for Computational Linguistics · cited 125,001 times · AI summary available
BERT introduces the bidirectional pretrained model whose un-fine-tuned relational knowledge the paper evaluates.
Abstract
We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powe…
-
Deep Contextualized Word RepresentationsMatthew E. Peters, Mark Neumann, M. Iyyer et al. · 2018 · North American Chapter of the Association for Computational Linguistics · cited 12,472 times · AI summary available
Deep Contextualized Word Representations presents pretrained bidirectional language-model representations used as groundwork for studying knowledge in contextual models.
Abstract
We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis. We also present an analysis showing that exposing the deep internals of the pre…
-
T-REx: A Large Scale Alignment of Natural Language with Knowledge Base TriplesHady ElSahar, P. Vougiouklis, Arslen Remaci et al. · 2018 · International Conference on Language Resources and Evaluation · cited 396 times · Not summarized yet
T-REx provides alignments between Wikipedia text and Wikidata triples, a dataset resource for studying relations expressed in text.
Abstract
Alignments between natural language and Knowledge Base (KB) triples are an essential prerequisite for training machine learning approaches employed in a variety of Natural Language Processing problems. These include Relation Extraction, KB Population, Question Answering and Natural Language Generation from KB triples. Available datasets that provide those alignments are plagued by significant shortcomings – they are of limited size, they exhibit a restricted predicate coverage, and/or they are of unreported quality. To alleviate these shortcomings, we present T-REx, a dataset of large scale alignments between Wikipedia abstracts and Wikidata triples. T-REx consists of 11 million triples alig…
-
Dissecting Contextual Word Embeddings: Architecture and RepresentationMatthew E. Peters, Mark Neumann, Luke Zettlemoyer et al. · 2018 · Conference on Empirical Methods in Natural Language Processing · cited 508 times · Not summarized yet
Dissecting Contextual Word Embeddings studies how language-model architectures affect learned contextual representations, rather than probing stored facts.
Abstract
Contextual word representations derived from pre-trained bidirectional language models (biLMs) have recently been shown to provide significant improvements to the state of the art for a wide range of NLP tasks. However, many questions remain as to how and why these models are so effective. In this paper, we present a detailed empirical study of how the choice of neural architecture (e.g. LSTM, CNN, or self attention) influences both end task accuracy and qualitative properties of the representations that are learned. We show there is a tradeoff between speed and accuracy, but all architectures learn high quality contextual representations that outperform word embeddings for four challenging …
-
A Survey of Reinforcement Learning Informed by Natural LanguageJelena Luketina, Nantas Nardelli, Gregory Farquhar et al. · 2019 · International Joint Conference on Artificial Intelligence · cited 319 times · Not summarized yet
A Survey of Reinforcement Learning Informed by Natural Language reviews how language understanding and textual knowledge support reinforcement learning.
Abstract
To be successful in real-world tasks, Reinforcement Learning (RL) needs to exploit the compositional, relational, and hierarchical structure of the world, and learn to transfer it to the task at hand. Recent advances in representation learning for language make it possible to build models that acquire world knowledge from text corpora and integrate this knowledge into downstream decision making problems. We thus argue that the time is right to investigate a tight integration of natural language understanding into RL in particular. We survey the state of the field, including work on instruction following, text games, and learning from textual domain knowledge. Finally, we call for the develop…
Papers that cite it
-
Probing for Knowledge Attribution in Large Language ModelsIvo Brink, A. Boer, Dennis Ulmer · 2026 · Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP · cited 1 time · Not summarized yet
Probing for Knowledge Attribution in Large Language Models identifies whether outputs rely on internal knowledge or supplied context, examining factual knowledge use.
Abstract
Large language model (LLM) hallucinations, meaning fluent but factually incorrect generations, fall into two types: faithfulness violations, where the model misuses provided context, and factuality violations, where answers reflect errors in internal knowledge. Proper mitigation depends on knowing which source drives each answer. We study contributive attribution, i.e. the classification of the dominant knowledge source behind each output, and show that a simple linear probe trained on hidden representations can reliably identify it. We introduce AttriWiki, a self-supervised pipeline that automatically generates labelled training data by prompting models to recall withheld entities from memo…
-
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor K. W. Shin, Yasaman Razeghi, Robert L. Logan et al. · 2020 · Conference on Empirical Methods in Natural Language Processing (EMNLP) · cited 1,304 times · Not summarized yet
AutoPrompt automatically searches for cloze prompts to elicit knowledge and task abilities from pretrained language models.
Abstract
The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining.Reformulating tasks as fillin-the-blanks problems (e.g., cloze tests) is a natural approach for gauging such knowledge, however, its usage is limited by the manual effort and guesswork required to write suitable prompts.To address this, we develop AUTOPROMPT, an automated method to create prompts for a diverse set of tasks, based on a gradient-guided search.Using AUTO-PROMPT, we show that masked language models (MLMs) have an inherent capability to perform sentiment analysis and natural language inference without additional parameters or finetuning, so…
-
Implicit Reasoning for Large Language Model-based Generative RecommendationYin-Han He, Liam Collins, B. Kumar et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 6 times · Not summarized yet
Implicit Reasoning for Large Language Model-based Generative Recommendation investigates how pretrained world knowledge can be elicited for recommendation.
Abstract
Large Language Models (LLMs) are increasingly adopted as backbones for Generative Recommendation (GR), promising access to pretrained world knowledge. Yet reliably invoking this knowledge for GR remains poorly understood. A key obstacle is that LLM-based GR typically represents items with Semantic IDs (SIDs), disrupting LLMs'natural-language reasoning interface because these tokens are unseen by the LLM during pretraining. Existing approaches address this with expensive multi-stage pipelines that ground SIDs and elicit explicit rationales, but offer limited insight into when and why each stage is necessary. In this work, we systematically decompose explicit reasoning training pipelines for L…
-
KARLA: Knowledge-base Augmented Retrieval for Language ModelsFrancois Crespin, Fabian M. Suchanek, Nils Holzenberger · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet
KARLA retrieves facts from a knowledge base during generation, grounding language-model outputs externally instead of relying solely on parametric knowledge.
Abstract
We propose a new method that allows an LLM to automatically pull in factual knowledge from a knowledge base during token generation. This means that (1)~factual knowledge in the LLM output can be updated without retraining the LLM, (2)~facts in the LLM output can be traced to the knowledge base for transparency and explainability, and (3)~smaller models can achieve the same factual accuracy as larger models. Our core idea is to train the model to produce special tokens that trigger a query to the knowledge base. Our experiments show that our method improves factual grounding in both short and long-form generation, and allows factual revisions to take effect through KB edits rather than param…
-
Co-Evolving Structured Knowledge and Reasoning in Language ModelsRyan Thomas Noonan, Lin-Xi Zhao, Meng-Han Xu et al. · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet
Co-Evolving Structured Knowledge and Reasoning in Language Models jointly builds a structured knowledge base and reasons over it for question answering.
Abstract
Retrieval-augmented methods improve factual accuracy by grounding language models in external knowledge, but retrieving over unstructured text often introduces irrelevant context and offers limited control over the retrieved information. Structured knowledge bases offer a more controllable alternative, yet they are expensive to construct and often brittle to reason over. To address these limitations, we propose KBevo: a co-evolving framework that jointly learns to construct a structured knowledge base and reason over it for knowledge-intensive question answering. By optimizing both components end-to-end with QA outcome rewards, our method enables reasoning success to directly improve the qua…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 4 November 2026.
The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.