# Simple Open-Vocabulary Object Detection with Vision Transformers

**Authors:** Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, Neil Houlsby  
**arXiv:** [2205.06230](https://arxiv.org/abs/2205.06230)  
**Date:** 2022-05-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2205.06230v1/

## Key points

- The paper proposes a recipe for transferring image-text models to open-vocabulary object detection
- Simple architectures with large-scale pre-training have led to significant improvements in image classification, but pre-training and scaling approaches for object detection are less well established
- The authors use a standard Vision Transformer architecture with minimal modifications, contrastive image-text pre-training, and end-to-end detection fine-tuning
- Increasing image-level pre-training and model size yield consistent improvements on the downstream detection task
- Adaptation strategies and regularizations are provided to attain very strong performance on zero-shot text conditioned and one shot image conditioned object detection
- The proposed approach achieves state of the art results on several benchmark datasets including COCO, LVIS, and Objects365
- It demonstrates its effectiveness in low resource settings by achieving competitive performance on the OpenImages dataset with only 10% of its training data
- Code and models are available on GitHub for reproducibility purposes.

## Summary

The paper "Simple Open-Vocabulary Object Detection with Vision Transformers" proposes a recipe for transferring image-text models to open-vocabulary object detection. The authors note that while combining simple architectures with large-scale pre-training has led to significant improvements in image classification, pre-training and scaling approaches for object detection are less well established, particularly in the long-tailed and open-vocabulary setting where training data is relatively scarce. To address this gap, the authors use a standard Vision Transformer architecture with minimal modifications, contrastive image-text pre-training, and end-to-end detection fine-tuning. The analysis of the scaling properties of this setup reveals that increasing image-level pre-training and model size yield consistent improvements on the downstream detection task. Furthermore, the authors provide adaptation strategies and regularizations needed to attain very strong performance on zero-shot text conditioned and one shot image conditioned object detection. The proposed approach achieves state of the art results on several benchmark datasets including COCO, LVIS, and Objects365. Additionally, it demonstrates its effectiveness in low resource settings by achieving competitive performance on the OpenImages dataset with only 10% of its training data. Overall, this paper presents a robust recipe for transferring image text models to open vocabulary object detection that achieves state of the art results while requiring minimal modifications to existing architectures. Code and models are available on GitHub for reproducibility purposes.