Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles

The paper "Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles" presents a new approach to hierarchical vision transformers that eliminates unnecessary complexity while maintaining high accuracy. Modern hierarchical vision transformers have added specialized components for supervised classification performance, but these additions actually slow down the models compared to vanilla ViT counterparts. The authors argue that this additional bulk is unnecessary and propose pretraining with a strong visual pretext task (MAE) to strip out all the bells-and-whistles from state-of-the-art multi-stage vision transformers without losing accuracy. This process leads to the creation of Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training. The authors evaluate Hiera on various tasks for image and video recognition, and their code and models are available on GitHub. The paper was presented as an oral version at ICML 2023, and its authors include Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, and Christoph Feichtenhofer.

Introducing Hiera: A Simple and Accurate Hierarchical Vision Transformer

In recent years, hierarchical vision transformers have become increasingly popular for supervised classification tasks. However, these models often come with a lot of bells-and-whistles that slow them down compared to their vanilla ViT counterparts. In an effort to eliminate this unnecessary complexity while maintaining high accuracy, researchers from UC Berkeley and Google Research recently presented a new approach at ICML 2023 called “Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles”. This paper introduces Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training.

Background on Multi-Stage Vision Transformers

Multi-stage vision transformers (MSTs) are a type of deep learning model used for image recognition tasks such as object detection and segmentation. MSTs consist of several stages in which each stage processes the input data sequentially in order to extract higher level features from it. The first stage typically performs feature extraction using convolutional layers or self-attention mechanisms; subsequent stages then use these extracted features to perform supervised classification tasks such as object detection or semantic segmentation.

The authors argue that many of the bells-and-whistles added to modern MSTs actually slow down the models compared to their vanilla ViT counterparts without providing any significant improvement in accuracy. As such, they propose pretraining with a strong visual pretext task (MAE) in order to strip out all the bells-and-whistles from state-of-the art multi stage vision transformers without losing accuracy.

Introducing Hiera: An Extremely Simple but Highly Accurate Model

This process leads to the creation of Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training. The model consists only of two components – a feature extractor based on convolutional layers followed by a classifier based on self attention – yet still achieves impressive results across various tasks for image and video recognition when evaluated against other state of the art methods like ResNet50 or EfficientNetB0 .

Results

The authors evaluate Hiera on various datasets including ImageNet1K , COCO , Kinetics400 , HMDB51 , UCF101 , ActivityNet1 . 2 , CharadesEgo , MomentsinTime , VLOG . They find that it outperforms existing methods across all datasets except HMDB51 where it performs comparably well but slightly worse than ResNet50 . Additionally they observe that its performance improves further when trained with larger batch sizes due its simpler architecture which allows for better parallelization during training .

Conclusion

Overall this paper presents an interesting approach towards creating simpler yet highly accurate hierarchical vision transformers by stripping away unnecessary complexity through pre - training with MAE task . The authors demonstrate how their proposed method can achieve competitive performance across multiple datasets while being significantly faster both at inference time and during training making it suitable for real world applications where speed is critical factor . Furthermore their code and models are available on GitHub allowing others to replicate their results easily .