# BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

**Authors:** Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova  
**arXiv:** [1810.04805](https://arxiv.org/abs/1810.04805)  
**Date:** 2018-10-11  
**Summary page:** https://www.summarizepaper.com/en/paper/1810.04805v1/

## Key points

- BERT is a new language representation model designed for pre-training deep bidirectional representations by conditioning on both left and right context in all layers.
- Pre-trained BERT representations can be fine-tuned with just one additional output layer to create state-of-the-art models for various tasks such as question answering and language inference without requiring significant task-specific architecture modifications.
- BERT achieves new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE benchmark to 80.4% (7.6% absolute improvement), MultiNLI accuracy to 86.7 (5.6% absolute improvement), and the SQuAD v1.1 question answering Test F1 to 93.2 (1.5% absolute improvement).
- These results outperform human performance by 2%.
- BERT's success can be attributed to its ability to capture contextual nuances in language better than previous models by leveraging bidirectional training at scale using transformers.
- BERT's pre-training approach enables it to learn from large amounts of unlabeled text data, making it more versatile and adaptable across different domains.
- This paper presents a significant advancement in natural language processing technology with potential applications in various fields such as chatbots, virtual assistants, and machine translation systems.

## Summary

In their paper titled "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova introduce a new language representation model called BERT. Unlike recent language models, BERT is designed to pre-train deep bidirectional representations by conditioning on both left and right context in all layers. This allows the pre-trained BERT representations to be fine-tuned with just one additional output layer to create state-of-the-art models for various tasks such as question answering and language inference without requiring significant task-specific architecture modifications. BERT is conceptually simple yet empirically powerful. It achieves new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE benchmark to 80.4% (7.6% absolute improvement), MultiNLI accuracy to 86.7 (5.6% absolute improvement), and the SQuAD v1.1 question answering Test F1 to 93.2 (1.5% absolute improvement). These results outperform human performance by 2%. The authors explain that BERT's success can be attributed to its ability to capture contextual nuances in language better than previous models by leveraging bidirectional training at scale using transformers. Additionally, they note that BERT's pre-training approach enables it to learn from large amounts of unlabeled text data, making it more versatile and adaptable across different domains. Overall, this paper presents a significant advancement in natural language processing technology with potential applications in various fields such as chatbots, virtual assistants, and machine translation systems.