# MetaFormer Is Actually What You Need for Vision

**Authors:** Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, Shuicheng Yan  
**arXiv:** [2111.11418](https://arxiv.org/abs/2111.11418)  
**Date:** 2021-11-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2111.11418v3/

## Key points

- Transformers have shown great potential in computer vision tasks
- Recent works challenge the belief that attention-based token mixer module is key to Transformers' competence
- Attention module can be replaced by spatial MLPs while still achieving competitive performance
- The general architecture of Transformers is more essential to performance than the specific token mixer module
- Replacing attention module with spatial pooling operator creates PoolFormer, which achieves competitive performance on multiple computer vision tasks
- PoolFormer achieves an impressive top-1 accuracy of 82.1% on ImageNet-1K dataset, surpassing other baselines with fewer parameters and MACs
- Introduction of MetaFormer, a general architecture abstracted from Transformers without specifying token mixer module
- MetaFormer plays a crucial role in achieving superior results for recent Transformer and MLP-like models in vision tasks
- Future research should focus on improving MetaFormer instead of solely focusing on token mixer modules

## Summary

In recent years, Transformers have shown great potential in computer vision tasks. It has been commonly believed that their attention-based token mixer module is the key contributor to their competence. However, recent works have challenged this belief by demonstrating that the attention module in Transformers can be replaced by spatial MLPs (Multi-Layer Perceptrons) while still achieving competitive performance. Building on this observation, the authors of this study propose a hypothesis that the general architecture of Transformers, rather than the specific token mixer module, is more essential to the model's performance. To test this hypothesis, they deliberately replace the attention module in Transformers with a simple spatial pooling operator for basic token mixing. Surprisingly, they find that the resulting model, called PoolFormer, achieves competitive performance on multiple computer vision tasks. For instance, on ImageNet-1K dataset, PoolFormer achieves an impressive top-1 accuracy of 82.1%. This surpasses well-tuned Vision Transformer/MLP-like baselines DeiT-B/ResMLP-B24 by 0.3%/1.1% accuracy while using significantly fewer parameters and MACs (Multiply-Accumulate operations). Specifically, PoolFormer achieves these results with 35%/52% fewer parameters and 50%/62% fewer MACs compared to DeiT-B/ResMLP-B24 models. The effectiveness of PoolFormer validates their hypothesis and leads them to introduce the concept of "MetaFormer", which is a general architecture abstracted from Transformers without specifying the token mixer module. Through extensive experiments and analysis, they argue that MetaFormer plays a crucial role in achieving superior results for recent Transformer and MLP-like models in vision tasks. This work highlights the need for future research dedicated to improving MetaFormer instead of solely focusing on token mixer modules. Additionally, PoolFormer serves as a starting baseline for designing future MetaFormer architectures. Overall, this study provides valuable insights into the importance of the general architecture of Transformers in computer vision tasks and suggests avenues for further advancements in this field.