# Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles

**Authors:** Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, Christoph Feichtenhofer  
**arXiv:** [2306.00989](https://arxiv.org/abs/2306.00989)  
**Date:** 2023-06-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.00989v1/

## Key points

- The paper presents a new approach to hierarchical vision transformers that eliminates unnecessary complexity while maintaining high accuracy.
- Specialized components added to modern hierarchical vision transformers for supervised classification performance slow down the models compared to vanilla ViT counterparts.
- Pretraining with a strong visual pretext task (MAE) is proposed to strip out all the bells-and-whistles from state-of-the-art multi-stage vision transformers without losing accuracy.
- Hiera is created, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training.
- Hiera was evaluated on various tasks for image and video recognition.
- Code and models for Hiera are available on GitHub.
- The paper was presented as an oral version at ICML 2023, and its authors include Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, and Christoph Feichtenhofer.

## Summary

The paper "Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles" presents a new approach to hierarchical vision transformers that eliminates unnecessary complexity while maintaining high accuracy. Modern hierarchical vision transformers have added specialized components for supervised classification performance, but these additions actually slow down the models compared to vanilla ViT counterparts. The authors argue that this additional bulk is unnecessary and propose pretraining with a strong visual pretext task (MAE) to strip out all the bells-and-whistles from state-of-the-art multi-stage vision transformers without losing accuracy. This process leads to the creation of Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training. The authors evaluate Hiera on various tasks for image and video recognition, and their code and models are available on GitHub. The paper was presented as an oral version at ICML 2023, and its authors include Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, and Christoph Feichtenhofer.