# Improved Multiscale Vision Transformers for Classification and Detection

**Authors:** Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, Christoph Feichtenhofer  
**arXiv:** [2112.01526](https://arxiv.org/abs/2112.01526)  
**Date:** 2021-12-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2112.01526v1/

## Key points

- Authors: Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, Christoph Feichtenhofer
- Study focuses on Multiscale Vision Transformers (MViT) for image and video classification and object detection
- Enhanced MViT version with decomposed relative positional embeddings and residual pooling connections to improve performance
- Implemented in five different sizes and evaluated on tasks like ImageNet classification, COCO object detection, Kinetics video recognition
- Improved MViT outperforms previous methods in accuracy
- Comparison between MViTs' pooling attention mechanism and window attention mechanisms shows pooling attention surpasses in accuracy/compute trade-off
- Achieves state-of-the-art performance: 88.8% accuracy on ImageNet classification, 56.1 box AP on COCO object detection, 86.1% accuracy on Kinetics-400 video classification
- Code and models publicly available for further research and development

## Summary

In the paper titled "Improved Multiscale Vision Transformers for Classification and Detection," authors Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer explore the use of Multiscale Vision Transformers (MViT) as a comprehensive architecture for image and video classification and object detection. The study introduces an enhanced version of MViT that incorporates decomposed relative positional embeddings and residual pooling connections to improve performance. The architecture is implemented in five different sizes and evaluated across various tasks including ImageNet classification, COCO object detection, and Kinetics video recognition. Results show that the improved MViT outperforms previous methods in terms of accuracy. One key aspect of the study is the comparison between MViTs' pooling attention mechanism and window attention mechanisms. It is found that MViTs' pooling attention surpasses window attention mechanisms in terms of accuracy/compute trade-off. Despite being presented without additional embellishments or complexities, MViT demonstrates state-of-the-art performance across three domains: achieving 88.8% accuracy on ImageNet classification, 56.1 box AP on COCO object detection, and 86.1% accuracy on Kinetics-400 video classification. The authors also emphasize their commitment to transparency by making both code and models publicly available for further research and development in the field. This study contributes valuable insights into enhancing Multiscale Vision Transformers for image and video analysis tasks, showcasing its potential for advancing performance benchmarks in these domains.