# Deep contextualized word representations

**Authors:** Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer  
**arXiv:** [1802.05365](https://arxiv.org/abs/1802.05365)  
**Date:** 2018-02-15  
**Summary page:** https://www.summarizepaper.com/en/paper/1802.05365v1/

## Key points

- The paper introduces a novel approach to word representation that captures syntax, semantics, and variations in word use across different linguistic contexts.
- Deep bidirectional language models (biLMs) are used to learn word vectors based on the internal states of the model.
- The biLMs are pre-trained on a large text corpus.
- Contextualized word representations can be easily integrated into existing models and improve performance in various NLP tasks.
- State-of-the-art performance is achieved in six NLP problems, including question answering, textual entailment, and sentiment analysis.
- Exposing the deep internals of the pre-trained network allows downstream models to effectively leverage different types of semi-supervision signals.
- The method outperforms existing approaches in several NLP tasks, emphasizing the importance of considering both syntactic and semantic aspects of word usage while accounting for variations across different linguistic contexts.

## Summary

The paper titled "Deep contextualized word representations" introduces a novel approach to word representation that captures both the complex characteristics of word use, such as syntax and semantics, and the variations in these uses across different linguistic contexts. The authors propose using deep bidirectional language models (biLMs) to learn word vectors based on the internal states of the model. These biLMs are pre-trained on a large text corpus. The authors demonstrate that their contextualized word representations can be easily integrated into existing models and yield significant improvements across various challenging natural language processing (NLP) tasks. Specifically, they achieve state-of-the-art performance in six NLP problems, including question answering, textual entailment, and sentiment analysis. Furthermore, an analysis is presented highlighting the importance of exposing the deep internals of the pre-trained network which allows downstream models to leverage different types of semi-supervision signals effectively. Overall, this research introduces a powerful method for generating deep contextualized word representations that outperform existing approaches in several NLP tasks. The findings emphasize the significance of considering both syntactic and semantic aspects of word usage while accounting for variations across different linguistic contexts.