# Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

**Authors:** Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang  
**arXiv:** [2303.05499](https://arxiv.org/abs/2303.05499)  
**Date:** 2023-03-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.05499v1/

## Key points

- Authors present a novel open-set object detector called Grounding DINO
- Grounding DINO combines DINO with grounded pre-training for object detection using human inputs
- Proposed approach includes feature enhancer, language-guided query selection mechanism, and cross-modality decoder for fusion
- Evaluation includes referring expression comprehension for objects specified with attributes
- Grounding DINO performs well on COCO, LVIS, ODinW, and RefCOCO/+/g datasets
- Achieves impressive results on zero-shot transfer benchmarks without training data from COCO
- Code for implementing Grounding DINO will be available at the provided URL
- Demonstrates significant advancements in open set object detection by incorporating language and vision modalities

## Summary

In this paper titled "Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection," authors Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu and Lei Zhang present a novel open-set object detector called Grounding DINO. This detector combines the Transformer-based detector DINO with grounded pre-training to detect arbitrary objects using human inputs such as category names or referring expressions. The key challenge in open-set object detection is generalizing concepts from closed-set detectors to handle open-set scenarios. To address this challenge and effectively fuse language and vision modalities, the authors propose a three-phase approach for conceptually dividing a closed-set detector. The proposed approach includes a feature enhancer, a language-guided query selection mechanism and a cross-modality decoder for cross-modality fusion. While previous works mainly focus on evaluating open set object detection on novel categories, the authors propose an additional evaluation on referring expression comprehension for objects specified with attributes. The proposed Grounding DINO model performs remarkably well on all three settings and achieves impressive results on benchmark datasets such as COCO (Common Objects in Context), LVIS (Large Vocabulary Instance Segmentation), ODinW (Open Domain Object Detection in the Wild) and RefCOCO/+/g (ReferItGame). Notably, Grounding DINO achieves an impressive average precision (AP) of $52.5$ on the COCO detection zero shot transfer benchmark without any training data from COCO. It also sets a new record on the ODinW zero shot benchmark with a mean AP of $26.1$. The code for implementing Grounding DINO will be made available at \url{https://github.com/IDEA Research/GroundingDINO}. Overall, the proposed Grounding DINO model demonstrates significant advancements in open set object detection by effectively incorporating language and vision modalities. The impressive performance on various benchmark datasets showcases its potential for real world applications in object detection tasks.