# Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth

**Authors:** Yihe Dong, Jean-Baptiste Cordonnier, Andreas Loukas  
**arXiv:** [2103.03404](https://arxiv.org/abs/2103.03404)  
**Date:** 2021-03-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2103.03404v1/

## Key points

- The paper discusses the effectiveness of attention-based architectures in machine learning.
- The authors propose a new way to understand self-attention networks by decomposing their output into smaller terms involving the operation of attention heads across layers.
- Self-attention has a strong inductive bias towards "token uniformity", which can cause output convergence to a rank-1 matrix without skip connections or multi-layer perceptrons (MLPs).
- Skip connections and MLPs prevent output degeneration.
- Experiments were conducted on different variants of standard transformer architectures to verify identified convergence phenomena and study the effects of path length on performance in three tasks: memorization, sorting, and convex hull.
- Short paths carry predictive power with accuracy above 0.8, 0.6, and 0.65 in the respective tasks while longer paths do not perform much better than random guessing.
- Length zero paths contain no useful information about the task.
- The models used for the experiments had varying depths (L), number of heads (H), and hidden dimensions (d).

## Summary

This paper addresses the effectiveness of attention-based architectures in machine learning and proposes a new way to understand self-attention networks. The authors show that the output of these networks can be decomposed into smaller terms, each involving the operation of a sequence of attention heads across layers. Using this decomposition, they prove that self-attention possesses a strong inductive bias towards "token uniformity". Specifically, without skip connections or multi-layer perceptrons (MLPs), the output converges doubly exponentially to a rank-1 matrix. However, skip connections and MLPs prevent the output from degenerating. The authors conduct experiments on different variants of standard transformer architectures to verify their identified convergence phenomena. They also study the effects of path length on performance in three tasks: memorization, sorting, and convex hull. The results show that short paths carry predictive power with accuracy above 0.8, 0.6, and 0.65 in the respective tasks while longer paths do not perform much better than random guessing. Length zero paths contain no useful information about the task. The depths (L), number of heads (H), and hidden dimensions (d) for the three models are: L:6, H:2, d :250 for memorization; L:6, H:2, d:48 for sorting; and L:6, H:3, d:84 for convex hull. Overall, this work provides insights into why attention-based architectures are effective and sheds light on how path length affects performance in certain tasks by conducting experiments on different variants of standard transformer architectures with varying depths (L), number of heads (H), and hidden dimensions (d). The results demonstrate that short paths carry predictive power while longer paths do not perform much better than random guessing and length zero paths contain no useful information about the task.