# An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

**Authors:** Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby  
**arXiv:** [2010.11929](https://arxiv.org/abs/2010.11929)  
**Date:** 2020-10-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2010.11929v1/

## Key points

- Transformer architecture is widely used in natural language processing tasks but not in computer vision.
- Attention is typically used with convolutional networks or to replace certain components of them in computer vision.
- A recent study by Google Research shows that a pure transformer applied directly to sequences of image patches can perform exceptionally well on image classification tasks.
- The Vision Transformer (ViT) uses a transformer-based model to process images as sequences of patches rather than pixels.
- ViT achieved excellent performance compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.
- This breakthrough could revolutionize computer vision by offering an alternative approach that does not rely on CNNs and could lead to faster training times and better accuracy for various computer vision applications.
- Transformers could be used more widely in computer vision tasks beyond just object recognition and classification.
- Fine-tuning code and pre-trained models are available on GitHub for those interested in exploring this new technology further.

## Summary

The Transformer architecture has become the go-to standard for natural language processing tasks, but its applications to computer vision have been limited. In vision, attention is typically used in conjunction with convolutional networks or to replace certain components of them while keeping their overall structure in place. However, a recent study by Alexey Dosovitskiy and his team at Google Research shows that this reliance on CNNs is not necessary. They demonstrate that a pure transformer applied directly to sequences of image patches can perform exceptionally well on image classification tasks. The researchers developed the Vision Transformer (ViT), which uses a transformer-based model to process images as sequences of patches rather than pixels. The ViT model was pre-trained on large amounts of data and then transferred to multiple mid-sized or small image recognition benchmarks such as ImageNet, CIFAR-100, and VTAB. The results showed that ViT achieved excellent performance compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train. This breakthrough could revolutionize computer vision by offering an alternative approach that does not rely on CNNs. The ViT model's success suggests that transformers could be used more widely in computer vision tasks beyond just object recognition and classification. This new approach could lead to faster training times and better accuracy for various computer vision applications. Fine-tuning code and pre-trained models are available on GitHub for those interested in exploring this new technology further.