# Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet

**Authors:** Luke Melas-Kyriazi  
**arXiv:** [2105.02723](https://arxiv.org/abs/2105.02723)  
**Date:** 2021-05-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2105.02723v1/

## Key points

- The study questions the necessity of attention layers in vision transformers
- Experiment involved replacing attention layer with feed-forward layer over patch dimension
- Modified architecture achieved impressive results on ImageNet
- Suggests other factors like patch embeddings and training procedures may be more important than attention
- Practical advantages of feed-forward-only model include linear complexity with respect to sequence length
- Raises questions about the importance of attention layers in transformer architectures

## Summary

In the study "Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet," the authors explore the necessity of attention layers in vision transformers. Vision transformers have gained popularity for their strong performance in image classification tasks, often attributed to their multi-head attention layers. However, the extent to which attention contributes to this performance remains unclear. The authors conduct experiments where they replace the attention layer in a vision transformer with a feed-forward layer applied over the patch dimension. This results in an architecture consisting of a series of feed-forward layers applied alternately over the patch and feature dimensions. Surprisingly, this modified architecture achieves impressive results on ImageNet, with a ViT/DeiT-base-sized model achieving 74.9% top-1 accuracy compared to 77.9% and 79.9% for traditional ViT and DeiT models, respectively. These findings suggest that aspects other than attention, such as patch embeddings and training procedures, may play a more significant role in the success of vision transformers than previously thought. The study also highlights practical advantages of using a feed-forward-only model, such as linear complexity with respect to sequence length compared to quadratic complexity in traditional vision transformers. While the feed-forward-only model may have limitations such as working only on fixed-length sequences, its performance raises questions about the importance of attention layers in transformer architectures. The results prompt further exploration into understanding why current models are effective and challenge existing assumptions about the key components driving their success in various vision tasks.