# Language Models as Knowledge Bases?

**Authors:** Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, Sebastian Riedel  
**arXiv:** [1909.01066](https://arxiv.org/abs/1909.01066)  
**Date:** 2019-09-03  
**Summary page:** https://www.summarizepaper.com/en/paper/1909.01066v1/

## Key points

- Recent progress in pretraining language models on large textual corpora has resulted in significant advancements in downstream NLP tasks
- Language models learn linguistic knowledge and store relational knowledge from the training data
- Language models offer advantages over structured knowledge bases, such as no requirement for schema engineering, ability to query about an open class of relations, easy scalability to more data, and no need for human supervision during training
- BERT exhibits competitive relational knowledge compared to traditional NLP methods even without fine-tuning
- BERT performs remarkably well on open-domain question answering compared to a supervised baseline
- Certain types of factual knowledge are learned more readily than others through standard language model pretraining approaches
- Pretrained language models have potential as unsupervised open-domain question answering systems
- The researchers provide code for reproducing their analysis at https://github.com/facebookresearch/LAMA

## Summary

Recent progress in pretraining language models on large textual corpora has resulted in significant advancements in downstream NLP tasks. These language models not only learn linguistic knowledge but also store relational knowledge present in the training data, enabling them to answer queries structured as "fill-in-the-blank" cloze statements. Compared to structured knowledge bases, language models offer several advantages such as no requirement for schema engineering, the ability to query about an open class of relations, easy scalability to more data, and no need for human supervision during training. In this study, the authors conduct a comprehensive analysis of the relational knowledge already present in a wide range of state-of-the-art pretrained language models without fine-tuning. They discover that BERT, one of the popular language models, exhibits competitive relational knowledge compared to traditional NLP methods that have access to oracle knowledge even without fine-tuning. Additionally, BERT performs remarkably well on open-domain question answering when compared against a supervised baseline. The authors also observe that certain types of factual knowledge are learned more readily than others through standard language model pretraining approaches. The impressive ability of these pretrained language models to recall factual knowledge without any fine-tuning showcases their potential as unsupervised open-domain question answering systems. The researchers provide code for reproducing their analysis which is available at https://github.com/facebookresearch/LAMA. Overall, this study highlights the valuable role that pretrained language models can play as knowledge bases by leveraging both linguistic and relational knowledge from large textual corpora. Their findings contribute to advancing natural language processing tasks and demonstrate the potential for further exploration and improvement in unsupervised question answering systems.