# What do Vision Transformers Learn? A Visual Exploration

**Authors:** Amin Ghiasi, Hamid Kazemi, Eitan Borgnia, Steven Reich, Manli Shu, Micah Goldblum, Andrew Gordon Wilson, Tom Goldstein  
**arXiv:** [2212.06727](https://arxiv.org/abs/2212.06727)  
**Date:** 2022-12-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.06727v1/

## Key points

- Vision Transformers (ViTs) are popular for computer vision tasks
- There is a lack of comprehensive understanding of how ViTs work and what they learn
- Neurons in ViTs trained with language model supervision are activated by semantic concepts rather than visual features
- ViTs and CNNs both detect image background features, but ViT predictions depend less on high-frequency information
- Both architecture types exhibit similar behavior in terms of feature progression from abstract patterns to concrete objects
- ViTs maintain spatial information in all layers except the final layer, which behaves as a learned global pooling operation and discards spatial information
- Large-scale visualizations were conducted on various ViT variants to validate the method's effectiveness
- This paper provides valuable insights into the workings of ViTs and sheds light on why they are becoming increasingly popular for computer vision tasks.

## Summary

In recent years, Vision Transformers (ViTs) have emerged as the go-to architecture for computer vision tasks. However, despite their growing popularity, we still lack a comprehensive understanding of how ViTs work and what they learn. While previous studies have explored the mechanisms of Convolutional Neural Networks (CNNs), visualizing ViTs remains challenging. In this paper, authors Amin Ghiasi et al. address these challenges and provide insights into the workings of ViTs. The authors begin by observing that neurons in ViTs trained with language model supervision are activated by semantic concepts rather than visual features. They also investigate the differences between ViTs and CNNs and find that both architectures detect image background features but that ViT predictions depend far less on high-frequency information. Additionally, both architecture types exhibit similar behavior in terms of feature progression from abstract patterns in early layers to concrete objects in late layers. The authors further demonstrate that ViTs maintain spatial information in all layers except the final layer, which behaves as a learned global pooling operation and discards spatial information. This finding contrasts with previous works that suggested otherwise. To validate their method's effectiveness, the authors conduct large-scale visualizations on various ViT variants such as DeiT, CoaT, ConViT, PiT, Swin, and Twin. The results show that their approach is effective at revealing important insights into how ViTs work. Overall, this paper provides valuable insights into the workings of ViTs and sheds light on why they are becoming increasingly popular for computer vision tasks.