# Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers

**Authors:** Dahun Kim, Anelia Angelova, Weicheng Kuo  
**arXiv:** [2305.07011](https://arxiv.org/abs/2305.07011)  
**Date:** 2023-05-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.07011v1/

## Key points

- Region-aware Open-vocabulary Vision Transformers (RO-ViT) is a contrastive image-text pretraining method
- RO-ViT aims to bridge the gap between image-level pretraining and open-vocabulary object detection
- The paper proposes randomly cropping and resizing regions of positional embeddings instead of using whole image positional embeddings in the pretraining phase
- Focal loss is used instead of softmax cross entropy loss in contrastive learning for better learning informative examples
- Recent advances in novel object proposals are leveraged to improve open-vocabulary detection finetuning
- RO-ViT achieves a state-of-the-art performance on LVIS benchmark, surpassing existing approaches by +5.8 points
- RO-ViT demonstrates competitive zero-shot transfer detection capabilities
- RO-ViT improves both open-vocabulary object detection and image-level representation
- RO-ViT achieves state-of-the-art results on 9 out of 12 metrics on COCO and Flickr image text retrieval benchmarks
- Ablation studies confirm the effectiveness of the proposed methods in contrastive image text pretraining

## Summary

The paper introduces Region-aware Open-vocabulary Vision Transformers (RO-ViT), a contrastive image-text pretraining method that aims to bridge the gap between image-level pretraining and open-vocabulary object detection. The authors propose a novel approach for the pretraining phase, where regions of positional embeddings are randomly cropped and resized instead of using the whole image positional embeddings. This modification aligns better with the use of positional embeddings at the region-level in the detection finetuning phase. In addition, the paper replaces the common softmax cross entropy loss in contrastive learning with focal loss, which helps in better learning informative yet difficult examples. The authors also leverage recent advances in novel object proposals to improve open-vocabulary detection finetuning. The full model is evaluated on two benchmark datasets: LVIS and COCO open-vocabulary detection benchmarks, as well as zero-shot transfer detection. RO-ViT achieves a state-of-the-art performance of 32.1 $AP_r$ on LVIS, surpassing the best existing approach by +5.8 points, and demonstrates competitive zero-shot transfer detection capabilities. Surprisingly, RO-ViT not only improves open-vocabulary object detection but also enhances image-level representation. It achieves state-of-the art results on 9 out of 12 metrics on COCO and Flickr image text retrieval benchmarks, outperforming competitive approaches that utilize larger models. Ablation studies are conducted to further analyze different aspects of RO ViT's performance. The results demonstrate the effectiveness of the proposed methods in contrastive image text pretraining. Overall, this work presents an innovative approach to address the challenges in open vocabulary object detection by leveraging region aware pretraining with Vision Transformers. The experimental results highlight its superiority over existing methods and its potential for improving both open vocabulary object detection and image text retrieval tasks.