# Teaching Matters: Investigating the Role of Supervision in Vision Transformers

**Authors:** Matthew Walmer, Saksham Suri, Kamal Gupta, Abhinav Shrivastava  
**arXiv:** [2212.03862](https://arxiv.org/abs/2212.03862)  
**Date:** 2022-12-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.03862v1/

## Key points

- Vision Transformers (ViTs) behavior under different learning paradigms
- Comparative analysis of ViTs trained with different supervision techniques
- Identification of Offset Local Attention Heads as a consistent behavior across all supervision methods
- ViTs' flexibility and adaptability in processing local and global information depending on training method
- Competitive features from contrastive self-supervised methods compared to explicitly supervised features
- Similarities between representations learned by reconstruction-based models and contrastive self-supervised models
- Varying optimal layer for a given task based on supervision method and specific task
- Insights into ViTs' behavior and ability to learn diverse behaviors while effectively processing information
- Effectiveness of contrastive self-supervised methods compared to explicit supervision for certain tasks.

## Summary

In their paper titled "Teaching Matters: Investigating the Role of Supervision in Vision Transformers," authors Matthew Walmer, Saksham Suri, Kamal Gupta, and Abhinav Shrivastava explore the behavior of Vision Transformers (ViTs) under different learning paradigms. ViTs have gained significant popularity in recent years and have been widely used in various applications. However, it remains unclear how their behavior varies based on different methods of supervision. The authors conduct a comparative analysis of ViTs trained using different supervision techniques and observe that these models exhibit a diverse range of behaviors in terms of attention mechanisms, representations, and downstream performance. Notably, they identify a consistent behavior across all supervision methods called Offset Local Attention Heads. These self-attention heads attend to a token adjacent to the current token with a fixed directional offset—an observation that has not been highlighted in prior research. The study demonstrates that ViTs possess high flexibility and adaptability by learning to process both local and global information in different orders depending on the training method employed. The authors find that contrastive self-supervised methods yield features that are competitive with explicitly supervised features and even outperform them for part-level tasks. Additionally, they discover non-trivial similarities between the representations learned by reconstruction-based models and contrastive self-supervised models. Furthermore, the authors investigate how the optimal layer for a given task varies based on both the supervision method used and the specific task at hand. This finding further emphasizes the varying order of information processing within ViTs. Overall, this study provides valuable insights into the behavior of ViTs under different learning paradigms. It highlights their ability to learn diverse behaviors while processing local and global information effectively. The findings also shed light on the effectiveness of contrastive self-supervised methods compared to explicit supervision for certain tasks.