AI summary · arXiv: 1909.01066v1 · cs.CL

Language Models as Knowledge Bases?

Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, Sebastian Riedel
Original paper · DOI 10.18653/v1/d19-1250 Submitted to arXiv on 03 Sep. 2019 Published in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (2019) accepted at EMNLP 2019 License: NONEXCLUSIVE-DISTRIB 1.0

How to read this summary? Illustrated guide

The essentials

  • Recent progress in pretraining language models on large textual corpora has resulted in significant advancements in downstream NLP tasks
  • Language models learn linguistic knowledge and store relational knowledge from the training data
  • Language models offer advantages over structured knowledge bases, such as no requirement for schema engineering, ability to query about an open class of relations, easy scalability to more data, and no need for human supervision during training
  • BERT exhibits competitive relational knowledge compared to traditional NLP methods even without fine-tuning
  • BERT performs remarkably well on open-domain question answering compared to a supervised baseline
  • Certain types of factual knowledge are learned more readily than others through standard language model pretraining approaches
  • Pretrained language models have potential as unsupervised open-domain question answering systems
  • The researchers provide code for reproducing their analysis at https://github.com/facebookresearch/LAMA

Key points written by AI: they may contain errors, check them against the paper.

AI-generated keywords: Pretrained Language Models Relational Knowledge BERT Factual Knowledge Unsupervised Question Answering

Read further

  • Comprehensive Summary
  • Layman's Summary
  • Blog article
  • Original abstract
Recent progress in pretraining language models on large textual corpora has resulted in significant advancements in downstream NLP tasks. These language models not only learn linguistic knowledge but also store relational knowledge present in the training data, enabling them to answer queries structured as "fill-in-the-blank" cloze statements. Compared to structured knowledge bases, language models offer several advantages such as no requirement for schema engineering, the ability to query about an open class of relations, easy scalability to more data, and no need for human supervision during training. In this study, the authors conduct a comprehensive analysis of the relational knowledge already present in a wide range of state-of-the-art pretrained language models without fine-tuning. They discover that BERT, one of the popular language models, exhibits competitive relational knowledge compared to traditional NLP methods that have access to oracle knowledge even without fine-tuning. Additionally, BERT performs remarkably well on open-domain question answering when compared against a supervised baseline. The authors also observe that certain types of factual knowledge are learned more readily than others through standard language model pretraining approaches. The impressive ability of these pretrained language models to recall factual knowledge without any fine-tuning showcases their potential as unsupervised open-domain question answering systems. The researchers provide code for reproducing their analysis which is available at https://github.com/facebookresearch/LAMA. Overall, this study highlights the valuable role that pretrained language models can play as knowledge bases by leveraging both linguistic and relational knowledge from large textual corpora. Their findings contribute to advancing natural language processing tasks and demonstrate the potential for further exploration and improvement in unsupervised question answering systems.

Was this synthesis useful?

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Figures and plots

Ask about a figure by number or use its question button to request visual analysis. Up to two figures and four image panels per question. Values read from plots are approximate; check the original data.

AI assistant instructions?

Tools for this paper

Explain it differently 1 free rewrite left today
Save to a collection Group papers in collections to compare them, chat with them and export them. Continue with Google Create a free account Log in

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero Zotero guide
Citation alerts 299 papers already cite it: be emailed each new one. Citation alerts guide
Who works on it?

Among the 31 papers around this one, the name that comes back most: Adam Roberts (2 papers)

3 more people sign several of them. The full map adds their institutions and approaches, from the papers of the last years.

Continue with Google Create a free account: your first map is free
Weekly letter The latest letter of its field, "Artificial intelligence this week" of 16 November 2026: Artificial intelligence this week: What impressive scores leave out · Receive one by email for free

Papers around this one

See them on a map

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

  1. Very close Builds on this paper Cites this paper
    How Much Knowledge Can You Pack into the Parameters of a Language Model?
    Adam Roberts, Colin A. Raffel, Noam Shazeer · 2020 · Conference on Empirical Methods in Natural Language Processing · cited 1,157 times · Not summarized yet

    How Much Knowledge Can You Pack into the Parameters of a Language Model? extends parametric knowledge probing by fine-tuning models for closed-book open-domain question answering.

    Abstract

    It has recently been observed that neural language models trained on unstructured text can implicitly store and retrieve knowledge using natural language queries. In this short paper, we measure the practical utility of this approach by fine-tuning pre-trained models to answer questions without access to any external context or knowledge. We show that this approach scales surprisingly well with model size and outperforms models that explicitly look up knowledge on the open-domain variants of Natural Questions and WebQuestions. To facilitate reproducibility and future work, we release our code and trained models.

  2. Very close Another approach to the problem Cites this paper
    REALM: Retrieval-Augmented Language Model Pre-Training
    Kelvin Guu, Kenton Lee, Zora Tung et al. · 2020 · International Conference on Machine Learning · cited 3,608 times · Not summarized yet

    REALM augments language-model pretraining with a retriever over documents, addressing factual knowledge access through retrieval rather than relying only on model parameters.

    Abstract

    Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language modeling a…

  3. Close Another approach to the problem
    How to Query Language Models?
    Leonard Adolphs, S. Dhuliawala, Thomas Hofmann · 2021 · arXiv.org · preprint, not peer reviewed · cited 17 times · Not summarized yet

    How to Query Language Models? uses example-based prompts to improve access to factual knowledge through cloze-style queries.

    Abstract

    Large pre-trained language models (LMs) are capable of not only recovering linguistic but also factual and commonsense knowledge. To access the knowledge stored in mask-based LMs, we can use cloze-style questions and let the model fill in the blank. The flexibility advantage over structured knowledge bases comes with the drawback of finding the right query for a certain information need. Inspired by human behavior to disambiguate a question, we propose to query LMs by example. To clarify the ambivalent question"Who does Neuer play for?", a successful strategy is to demonstrate the relation using another subject, e.g.,"Ronaldo plays for Portugal. Who does Neuer play for?". We apply this appro…

  4. Close Same method, another problem
    Language Models are Open Knowledge Graphs
    Chenguang Wang, Xiao Liu, D. Song · 2020 · arXiv.org · preprint, not peer reviewed · cited 152 times · Not summarized yet

    Language Models are Open Knowledge Graphs extracts relational knowledge from pretrained language models to construct knowledge graphs without human supervision.

    Abstract

    This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e.g., BERT, GPT-2/3), without human supervision. Popular KGs (e.g, Wikidata, NELL) are built in either a supervised or semi-supervised manner, requiring humans to create knowledge. Recent deep language models automatically acquire knowledge from large-scale corpora via pre-training. The stored knowledge has enabled the language models to improve downstream NLP tasks, e.g., answering questions, and writing code and articles. In this paper, we propose an unsupervised method to cast the knowledge contained within language models into KGs. We show that KGs are constructed with a single forward pass of the …

  5. Close Same question
    Structured Knowledge Grounding for Question Answering
    Yu-Jie Lu, Siqi Ouyang, Kai-Rui Zhou · 2022 · cited 3 times · Not summarized yet

    Structured Knowledge Grounding for Question Answering asks how language models can use relational knowledge for knowledge-based question answering.

    Abstract

    Can language models (LM) ground question-answering (QA) tasks in the knowledge base via inherent relational reasoning ability? While previous models that use only LMs have seen some success on many QA tasks, more recent methods include knowledge graphs (KG) to complement LMs with their more logic-driven implicit knowledge. However, effectively extracting information from structured data, like KGs, empowers LMs to remain an open question, and current models rely on graph techniques to extract knowledge. In this paper, we propose to solely leverage the LMs to combine the language and knowledge for knowledge based question-answering with flexibility, breadth of coverage and structured reasoning…

  6. Close Same question
    Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation
    Ruiyang Ren, Yuhao Wang, Yingqi Qu et al. · 2023 · Computation and Language · cited 214 times · AI summary available

    Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation studies factual knowledge in language models and its effect on open-domain QA.

    Abstract

    Knowledge-intensive tasks (e.g., open-domain question answering (QA)) require a substantial amount of factual knowledge and often rely on external information for assistance. Recently, large language models (LLMs) (e.g., ChatGPT), have demonstrated impressive prowess in solving a wide range of tasks with world knowledge, including knowledge-intensive tasks. However, it remains unclear how well LLMs are able to perceive their factual knowledge boundaries, particularly how they behave when incorporating retrieval augmentation. In this study, we present an initial analysis of the factual knowledge boundaries of LLMs and how retrieval augmentation affects LLMs on open-domain QA. Specially, we fo…

  7. Close Same question
    Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA?
    Cunxiang Wang, Pai Liu, Yue Zhang · 2021 · Annual Meeting of the Association for Computational Linguistics · cited 97 times · Not summarized yet

    Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA? evaluates whether pretrained models retain facts needed to answer questions without external context.

    Abstract

    Recent work has investigated the interesting question using pre-trained language models (PLMs) as knowledge bases for answering open questions. However, existing work is limited in using small benchmarks with high test-train overlaps. We construct a new dataset of closed-book QA using SQuAD, and investigate the performance of BART. Experiments show that it is challenging for BART to remember training facts in high precision, and also challenging to answer closed-book questions even if relevant knowledge is retained. Some promising directions are found, including decoupling the knowledge memorizing process and the QA finetune process, forcing the model to recall relevant knowledge when questi…

  8. Close Same question
    When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
    Alex Troy Mallen, Akari Asai, Victor Zhong et al. · 2022 · Annual Meeting of the Association for Computational Linguistics · cited 1,634 times · Not summarized yet

    When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories probes factual recall and compares it with retrieval augmentation on entity-centric QA.

    Abstract

    Despite their impressive performance on diverse tasks, large language models (LMs) still struggle with tasks requiring rich world knowledge, implying the difficulty of encoding a wealth of world knowledge in their parameters. This paper aims to understand LMs’ strengths and limitations in memorizing factual knowledge, by conducting large-scale knowledge probing experiments on two open-domain entity-centric QA datasets: PopQA, our new dataset with 14k questions about long-tail entities, and EntityQuestions, a widely used open-domain QA dataset. We find that LMs struggle with less popular factual knowledge, and that retrieval augmentation helps significantly in these cases. Scaling, on the oth…

Prior work it builds on

  1. Close Groundwork for this paper Used in depth by this paper A method it uses
    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
    J. Devlin, Ming-Wei Chang, Kenton Lee et al. · 2019 · North American Chapter of the Association for Computational Linguistics · cited 125,001 times · AI summary available

    BERT introduces the bidirectional pretrained model whose un-fine-tuned relational knowledge the paper evaluates.

    Abstract

    We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powe…

  2. Related Groundwork for this paper Used in depth by this paper A method it uses
    Deep Contextualized Word Representations
    Matthew E. Peters, Mark Neumann, M. Iyyer et al. · 2018 · North American Chapter of the Association for Computational Linguistics · cited 12,472 times · AI summary available

    Deep Contextualized Word Representations presents pretrained bidirectional language-model representations used as groundwork for studying knowledge in contextual models.

    Abstract

    We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis. We also present an analysis showing that exposing the deep internals of the pre…

  3. Related Shared data, tool or method Used in depth by this paper A method it uses
    T-REx: A Large Scale Alignment of Natural Language with Knowledge Base Triples
    Hady ElSahar, P. Vougiouklis, Arslen Remaci et al. · 2018 · International Conference on Language Resources and Evaluation · cited 396 times · Not summarized yet

    T-REx provides alignments between Wikipedia text and Wikidata triples, a dataset resource for studying relations expressed in text.

    Abstract

    Alignments between natural language and Knowledge Base (KB) triples are an essential prerequisite for training machine learning approaches employed in a variety of Natural Language Processing problems. These include Relation Extraction, KB Population, Question Answering and Natural Language Generation from KB triples. Available datasets that provide those alignments are plagued by significant shortcomings – they are of limited size, they exhibit a restricted predicate coverage, and/or they are of unreported quality. To alleviate these shortcomings, we present T-REx, a dataset of large scale alignments between Wikipedia abstracts and Wikidata triples. T-REx consists of 11 million triples alig…

  4. Related Broader context Used in depth by this paper A method it uses
    Dissecting Contextual Word Embeddings: Architecture and Representation
    Matthew E. Peters, Mark Neumann, Luke Zettlemoyer et al. · 2018 · Conference on Empirical Methods in Natural Language Processing · cited 508 times · Not summarized yet

    Dissecting Contextual Word Embeddings studies how language-model architectures affect learned contextual representations, rather than probing stored facts.

    Abstract

    Contextual word representations derived from pre-trained bidirectional language models (biLMs) have recently been shown to provide significant improvements to the state of the art for a wide range of NLP tasks. However, many questions remain as to how and why these models are so effective. In this paper, we present a detailed empirical study of how the choice of neural architecture (e.g. LSTM, CNN, or self attention) influences both end task accuracy and qualitative properties of the representations that are learned. We show there is a tradeoff between speed and accuracy, but all architectures learn high quality contextual representations that outperform word embeddings for four challenging …

  5. Close Review
    A Survey of Reinforcement Learning Informed by Natural Language
    Jelena Luketina, Nantas Nardelli, Gregory Farquhar et al. · 2019 · International Joint Conference on Artificial Intelligence · cited 319 times · Not summarized yet

    A Survey of Reinforcement Learning Informed by Natural Language reviews how language understanding and textual knowledge support reinforcement learning.

    Abstract

    To be successful in real-world tasks, Reinforcement Learning (RL) needs to exploit the compositional, relational, and hierarchical structure of the world, and learn to transfer it to the task at hand. Recent advances in representation learning for language make it possible to build models that acquire world knowledge from text corpora and integrate this knowledge into downstream decision making problems. We thus argue that the time is right to investigate a tight integration of natural language understanding into RL in particular. We survey the state of the field, including work on instruction following, text games, and learning from textual domain knowledge. Finally, we call for the develop…

Papers that cite it

  1. Close Same question Uses this paper in depth
    Probing for Knowledge Attribution in Large Language Models
    Ivo Brink, A. Boer, Dennis Ulmer · 2026 · Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP · cited 1 time · Not summarized yet

    Probing for Knowledge Attribution in Large Language Models identifies whether outputs rely on internal knowledge or supplied context, examining factual knowledge use.

    Abstract

    Large language model (LLM) hallucinations, meaning fluent but factually incorrect generations, fall into two types: faithfulness violations, where the model misuses provided context, and factuality violations, where answers reflect errors in internal knowledge. Proper mitigation depends on knowing which source drives each answer. We study contributive attribution, i.e. the classification of the dominant knowledge source behind each output, and show that a simple linear probe trained on hidden representations can reliably identify it. We introduce AttriWiki, a self-supervised pipeline that automatically generates labelled training data by prompting models to recall withheld entities from memo…

  2. Close Another approach to the problem
    AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
    Taylor K. W. Shin, Yasaman Razeghi, Robert L. Logan et al. · 2020 · Conference on Empirical Methods in Natural Language Processing (EMNLP) · cited 1,304 times · Not summarized yet

    AutoPrompt automatically searches for cloze prompts to elicit knowledge and task abilities from pretrained language models.

    Abstract

    The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining.Reformulating tasks as fillin-the-blanks problems (e.g., cloze tests) is a natural approach for gauging such knowledge, however, its usage is limited by the manual effort and guesswork required to write suitable prompts.To address this, we develop AUTOPROMPT, an automated method to create prompts for a diverse set of tasks, based on a gradient-guided search.Using AUTO-PROMPT, we show that masked language models (MLMs) have an inherent capability to perform sentiment analysis and natural language inference without additional parameters or finetuning, so…

  3. Close Same method, another problem Uses this paper in depth
    Implicit Reasoning for Large Language Model-based Generative Recommendation
    Yin-Han He, Liam Collins, B. Kumar et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 6 times · Not summarized yet

    Implicit Reasoning for Large Language Model-based Generative Recommendation investigates how pretrained world knowledge can be elicited for recommendation.

    Abstract

    Large Language Models (LLMs) are increasingly adopted as backbones for Generative Recommendation (GR), promising access to pretrained world knowledge. Yet reliably invoking this knowledge for GR remains poorly understood. A key obstacle is that LLM-based GR typically represents items with Semantic IDs (SIDs), disrupting LLMs'natural-language reasoning interface because these tokens are unseen by the LLM during pretraining. Existing approaches address this with expensive multi-stage pipelines that ground SIDs and elicit explicit rationales, but offer limited insight into when and why each stage is necessary. In this work, we systematically decompose explicit reasoning training pipelines for L…

  4. Close Another approach to the problem
    KARLA: Knowledge-base Augmented Retrieval for Language Models
    Francois Crespin, Fabian M. Suchanek, Nils Holzenberger · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet

    KARLA retrieves facts from a knowledge base during generation, grounding language-model outputs externally instead of relying solely on parametric knowledge.

    Abstract

    We propose a new method that allows an LLM to automatically pull in factual knowledge from a knowledge base during token generation. This means that (1)~factual knowledge in the LLM output can be updated without retraining the LLM, (2)~facts in the LLM output can be traced to the knowledge base for transparency and explainability, and (3)~smaller models can achieve the same factual accuracy as larger models. Our core idea is to train the model to produce special tokens that trigger a query to the knowledge base. Our experiments show that our method improves factual grounding in both short and long-form generation, and allows factual revisions to take effect through KB edits rather than param…

  5. Close Another approach to the problem
    Co-Evolving Structured Knowledge and Reasoning in Language Models
    Ryan Thomas Noonan, Lin-Xi Zhao, Meng-Han Xu et al. · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet

    Co-Evolving Structured Knowledge and Reasoning in Language Models jointly builds a structured knowledge base and reasons over it for question answering.

    Abstract

    Retrieval-augmented methods improve factual accuracy by grounding language models in external knowledge, but retrieving over unstructured text often introduces irrelevant context and offers limited control over the retrieved information. Structured knowledge bases offer a more controllable alternative, yet they are expensive to construct and often brittle to reason over. To address these limitations, we propose KBevo: a co-evolving framework that jointly learns to construct a structured knowledge base and reason over it for knowledge-intensive question answering. By optimizing both components end-to-end with QA outcome rewards, our method enables reasoning success to directly improve the qua…

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

Updated on 4 November 2026.

The illustrated guide to the papers around a paper

The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.