# Attention Is All You Need

**Authors:** Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin  
**arXiv:** [1706.03762](https://arxiv.org/abs/1706.03762)  
**Date:** 2017-06-12  
**Summary page:** https://www.summarizepaper.com/en/paper/1706.03762v7/

## Key points

- The paper introduces a network architecture called the Transformer based on attention mechanisms
- The Transformer eliminates the need for recurrence and convolutions in encoder-decoder models
- The Transformer outperformed existing models in machine translation tasks in terms of quality, parallelizability, and training time
- Achieved a BLEU score of 28.4 on English-to-German translation task, surpassing previous best results by over 2 BLEU points
- Achieved a state-of-the-art BLEU score of 41.8 on English-to-French translation task after training for only 3.5 days on eight GPUs
- Successfully applied the Transformer to English constituency parsing with both large and limited training data
- Represents a significant advancement in sequence transduction models by leveraging attention mechanisms exclusively

## Summary

The paper titled "Attention Is All You Need" introduces a new network architecture called the Transformer, which is based solely on attention mechanisms. Unlike existing models that rely on recurrent or convolutional neural networks in an encoder-decoder configuration, the Transformer eliminates the need for recurrence and convolutions entirely. The authors conducted experiments on two machine translation tasks and found that the Transformer models outperformed existing models in terms of quality while also being more parallelizable and requiring less training time. Specifically, their model achieved a BLEU score of 28.4 on the WMT 2014 English-to-German translation task, surpassing previous best results by over 2 BLEU points. On the WMT 2014 English-to-French translation task, their model achieved a state-of-the-art BLEU score of 41.8 after training for only 3.5 days on eight GPUs, which is significantly less than the training costs of previous top-performing models. Furthermore, the authors demonstrated that the Transformer generalizes well to other tasks by successfully applying it to English constituency parsing with both large and limited training data. Overall, the introduction of the Transformer architecture represents a significant advancement in sequence transduction models by leveraging attention mechanisms exclusively. The results obtained from machine translation tasks showcase its superiority in terms of performance and efficiency compared to traditional recurrent or convolutional neural network-based models.