# Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

**Authors:** Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang  
**arXiv:** [2303.05499](https://arxiv.org/abs/2303.05499)  
**Date:** 2023-03-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.05499v4/

## Key points

- Authors present a novel open-set object detector called Grounding DINO
- Grounding DINO combines Transformer-based detector DINO with grounded pre-training
- Enables detection of arbitrary objects using human inputs such as category names or referring expressions
- Proposes a three-phase approach and tight fusion solution for effective language and vision modalities fusion
- Evaluates open set object detection on novel categories and referring expression comprehension for objects specified with attributes
- Achieves impressive results on benchmarks such as COCO, LVIS, ODinW, and RefCOCO/+/g
- Impressive Average Precision (AP) score of 52.5 on COCO detection zero shot transfer benchmark without any training data from COCO
- Sets a new record on ODinW zero shot benchmark with mean AP score of 26.1
- Code available at https://github.com/IDEA-Research/GroundingDINO

## Summary

In this paper titled "Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection," authors Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu and Lei Zhang present a novel open-set object detector called Grounding DINO. This detector combines the Transformer-based detector DINO with grounded pre-training to enable the detection of arbitrary objects using human inputs such as category names or referring expressions. The key challenge in open-set object detection is generalizing concepts from closed-set detectors by introducing language. To effectively fuse language and vision modalities, the authors propose a three-phase approach and a tight fusion solution that includes a feature enhancer for improved visual features; a language guided query selection mechanism; and a cross-modality decoder for cross-modality fusion. While previous works mainly focus on evaluating open set object detection on novel categories only, the authors also propose evaluations on referring expression comprehension for objects specified with attributes. Grounding DINO performs exceptionally well across all three settings and achieves impressive results on benchmarks such as COCO (Common Objects in Context), LVIS (Large Vocabulary Instance Segmentation), ODinW (Open Domain Object Detection in the Wild) and RefCOCO/+/g (ReferItGame). Notably it achieves an impressive Average Precision (AP) score of 52.5 on the COCO detection zero shot transfer benchmark without any training data from COCO. It also sets a new record on the ODinW zero shot benchmark with a mean AP score of 26.1. The authors provide code for their method at https://github.com/IDEA-Research/GroundingDINO . Overall this paper introduces Grounding DINO as an effective open set object detector that leverages grounded pre training and language integration to detect arbitrary objects with human inputs while achieving state of the art results on various benchmarks demonstrating its effectiveness in detecting arbitrary objects with human inputs.