# Reformer: The Efficient Transformer

**Authors:** Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya  
**arXiv:** [2001.04451](https://arxiv.org/abs/2001.04451)  
**Date:** 2020-01-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2001.04451v1/

## Key points

- The paper introduces two techniques to enhance the efficiency of Transformer models: locality-sensitive hashing and reversible residual layers.
- Locality-sensitive hashing reduces the complexity of attention from O($L^2$) to O($L\log L$), improving computational efficiency.
- Reversible residual layers allow for storing activations only once during training, enhancing memory efficiency.
- The resulting model, called Reformer, achieves performance comparable to traditional Transformers while being more memory-efficient and faster when processing long sequences.
- Reformer addresses the challenge of high computational costs associated with training large Transformers and provides a more efficient solution for handling long sequences.

## Summary

. The paper titled "Reformer: The Efficient Transformer" by Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya introduces two techniques to enhance the efficiency of Transformer models. While large Transformer models have achieved state-of-the-art results on various tasks, training these models can be prohibitively expensive, particularly for long sequences. The first technique proposed in this paper is the replacement of dot-product attention with locality-sensitive hashing. This change reduces the complexity of attention from O($L^2$) to O($L\log L$), where $L$ represents the length of the sequence. By using locality-sensitive hashing, the authors are able to significantly improve the computational efficiency of Transformers. Additionally, the authors introduce reversible residual layers as an alternative to standard residuals. This modification allows for storing activations only once during the training process instead of $N$ times, where $N$ denotes the number of layers in the model. This approach further enhances memory efficiency. The resulting model, called Reformer, achieves performance comparable to traditional Transformer models while being much more memory-efficient and faster when processing long sequences compared to traditional Transformers. The Reformer model addresses the challenge of high computational costs associated with training large Transformers and provides a more efficient solution for handling long sequences. In summary, this paper presents two key techniques - locality-sensitive hashing and reversible residual layers - that contribute to improving the efficiency of Transformer models. The Reformer model demonstrates impressive performance while reducing memory requirements and speeding up processing time for long sequences compared to traditional Transformers.