# Attention Is All You Need

**Authors:** Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin  
**arXiv:** [1706.03762](https://arxiv.org/abs/1706.03762)  
**Date:** 2017-06-12  
**Summary page:** https://www.summarizepaper.com/en/paper/1706.03762v5/

## Key points

- Authors propose a new neural network architecture called the Transformer for sequence transduction tasks such as machine translation
- The Transformer is based solely on attention mechanisms and does not require recurrence or convolutions like traditional models
- The model outperforms existing models in terms of quality while being more parallelizable and requiring significantly less training time
- Achieves 28.4 BLEU on the WMT 2014 English-to-German translation task and establishes a new single-model state-of-the-art BLEU score of 41.8 on the WMT 2014 English-to-French translation task after training for only 3.5 days on eight GPUs
- Successfully applied to English constituency parsing both with large and limited training data sets

## Summary

In their paper "Attention Is All You Need," authors Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin propose a new neural network architecture called the Transformer for sequence transduction tasks such as machine translation. The Transformer is based solely on attention mechanisms and does not require recurrence or convolutions like traditional models that rely on complex recurrent or convolutional neural networks in an encoder-decoder configuration with an attention mechanism connecting them. The authors demonstrate through experiments on two machine translation tasks that the Transformer outperforms existing models in terms of quality while being more parallelizable and requiring significantly less training time. Specifically, the model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task and establishes a new single-model state-of-the-art BLEU score of 41.8 on the WMT 2014 English-to-French translation task after training for only 3.5 days on eight GPUs - a fraction of the training costs of other best-performing models from literature. Furthermore, they show that the Transformer generalizes well to other tasks by successfully applying it to English constituency parsing both with large and limited training data sets.