# BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

**Authors:** Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova  
**arXiv:** [1810.04805](https://arxiv.org/abs/1810.04805)  
**Date:** 2018-10-11  
**Summary page:** https://www.summarizepaper.com/en/paper/1810.04805v2/

## Key points

- BERT is a groundbreaking language representation model introduced by Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
- BERT stands for Bidirectional Encoder Representations from Transformers.
- BERT pre-trains deep bidirectional representations from unlabeled text by considering both left and right context in all layers.
- BERT achieves state-of-the-art performance on various natural language processing tasks with minimal task-specific modifications.
- By fine-tuning the pre-trained BERT model with just one additional output layer, highly accurate models can be created for tasks such as question answering and language inference.
- BERT demonstrates simplicity and effectiveness through impressive results on eleven different natural language processing tasks.
- It significantly improves the GLUE score to 80.5% (a 7.7% absolute improvement), MultiNLI accuracy to 86.7% (a 4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (a 1.5 point absolute improvement), and SQuAD v2.0 Test F1 to 83.1 (a 5.1 point absolute improvement).
- Overall, BERT represents a significant advancement in language representation models with the potential to revolutionize various natural language processing applications.

## Summary

In their paper titled "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," authors Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova introduce a groundbreaking language representation model called BERT. BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations from unlabeled text by considering both left and right context in all layers. Unlike previous language representation models, BERT achieves state-of-the-art performance on various natural language processing tasks with minimal task-specific modifications. By fine-tuning the pre-trained BERT model with just one additional output layer, researchers can create highly accurate models for tasks such as question answering and language inference. The simplicity and effectiveness of BERT are demonstrated through its impressive results on eleven different natural language processing tasks. For instance, it significantly improves the GLUE score to 80.5% (a 7.7% absolute improvement), MultiNLI accuracy to 86.7% (a 4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (a 1.5 point absolute improvement), and SQuAD v2.0 Test F1 to 83.1 (a 5.1 point absolute improvement). Overall, BERT represents a significant advancement in language representation models and has the potential to revolutionize various natural language processing applications.