# Transformers without Tears: Improving the Normalization of Self-Attention

**Authors:** Toan Q. Nguyen, Julian Salazar  
**arXiv:** [1910.05895](https://arxiv.org/abs/1910.05895)  
**Date:** 2019-10-14  
**Summary page:** https://www.summarizepaper.com/en/paper/1910.05895v1/

## Key points

- Authors Toan Q. Nguyen and Julian Salazar evaluate three normalization-centric changes to improve Transformer training:
- Implementation of pre-norm residual connections (PreNorm) and smaller initializations for warmup-free, validation-based training with large learning rates.
- Introduction of $\ell_2$ normalization with a single scale parameter (ScaleNorm) for faster training and better performance outcomes.
- Affirmation of normalizing word embeddings to a fixed length (FixNorm).
- Study focuses on low-resource translation pairs from TED Talks-based corpora, showing consistent convergence and an average improvement of +1.1 BLEU over state-of-the-art bilingual baselines.
- Notable achievement of a new high score of 32.8 BLEU on the IWSLT'15 English-Vietnamese translation task.
- Observations include sharper performance curves, more consistent gradient norms, and a linear relationship between activation scaling and decoder depth due to proposed modifications.
- In the high-resource setting represented by the WMT'14 English-German translation task, ScaleNorm and FixNorm remain competitive while PreNorm exhibits a degradation in performance.
- Research emphasizes the importance of normalization-centric adjustments in enhancing Transformer training processes and improving translation performance across different language pairs and resource levels.

## Summary

, , , , 

In their paper titled "Transformers without Tears: Improving the Normalization of Self-Attention," authors Toan Q. Nguyen and Julian Salazar evaluate three normalization-centric changes to enhance Transformer training. The first change involves the implementation of pre-norm residual connections (PreNorm) and smaller initializations, which enable warmup-free, validation-based training with large learning rates. The second change introduces $\ell_2$ normalization with a single scale parameter (ScaleNorm), aimed at facilitating faster training and achieving better performance outcomes. Additionally, the authors reaffirm the effectiveness of normalizing word embeddings to a fixed length (FixNorm). The study conducted by Nguyen and Salazar focuses on five low-resource translation pairs derived from TED Talks-based corpora. Their findings indicate that these proposed changes consistently lead to convergence, resulting in an average improvement of +1.1 BLEU over state-of-the-art bilingual baselines. Particularly noteworthy is the achievement of a new high score of 32.8 BLEU on the IWSLT'15 English-Vietnamese translation task. Furthermore, the authors observe sharper performance curves, more consistent gradient norms, and establish a linear relationship between activation scaling and decoder depth as a result of their proposed modifications. Surprisingly, in the high-resource setting represented by the WMT'14 English-German translation task, ScaleNorm and FixNorm maintain competitiveness while PreNorm shows a degradation in performance. Overall, this research highlights the significance of normalization-centric adjustments in enhancing Transformer training processes and achieving improved translation performance across various language pairs and resource levels.