# Simple Open-Vocabulary Object Detection with Vision Transformers

**Authors:** Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, Neil Houlsby  
**arXiv:** [2205.06230](https://arxiv.org/abs/2205.06230)  
**Date:** 2022-05-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2205.06230v2/

## Key points

- Combination of simple architectures and large-scale pre-training has revolutionized image classification
- Pre-training and scaling approaches are not well-established in object detection, especially in long-tailed and open-vocabulary settings
- Authors propose a robust methodology for transferring image-text models to open-vocabulary object detection
- Standard Vision Transformer architecture with minimal modifications is used, along with contrastive image-text pre-training and end-to-end detection fine-tuning
- Increasing image-level pre-training and model size consistently improves the downstream detection task
- Adaptation strategies and regularizations are provided for exceptional performance in zero-shot text-conditioned and one-shot image-conditioned object detection scenarios
- Paper presents a strong recipe for applying image-text models to open-vocabulary object detection tasks
- Effectiveness of combining Vision Transformers with contrastive pre-training and fine-tuning techniques is showcased
- Scaling up both pre-training data and model size leads to improved performance
- Code and models are available on GitHub for researchers interested in exploring these methods.

## Summary

, , , , 

In recent years, the combination of simple architectures and large-scale pre-training has revolutionized image classification. However, when it comes to object detection, the use of pre-training and scaling approaches is not as well-established. This is particularly true in the long-tailed and open-vocabulary setting where training data is limited. To address this gap, the authors propose a robust methodology for transferring image-text models to open-vocabulary object detection. They employ a standard Vision Transformer architecture with minimal modifications and leverage contrastive image-text pre-training followed by end-to-end detection fine-tuning. The authors conduct an extensive analysis of the scaling properties of this setup and demonstrate that increasing image-level pre-training and model size consistently leads to improvements in the downstream detection task. They also provide adaptation strategies and regularizations necessary to achieve exceptional performance in zero-shot text-conditioned and one-shot image-conditioned object detection scenarios. Overall, this paper presents a strong recipe for applying image-text models to open-vocabulary object detection tasks. The proposed methodology showcases the effectiveness of combining Vision Transformers with contrastive pre-training and fine-tuning techniques. The authors' thorough analysis highlights the benefits of scaling up both pre-training data and model size for improved performance. Researchers interested in exploring these methods can access the code and models on GitHub.