# Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation

**Authors:** Feng Li, Hao Zhang, Huaizhe xu, Shilong Liu, Lei Zhang, Lionel M. Ni, Heung-Yeung Shum  
**arXiv:** [2206.02777](https://arxiv.org/abs/2206.02777)  
**Date:** 2022-06-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2206.02777v1/

## Key points

- Mask DINO is a transformer-based framework for object detection and segmentation
- It extends DINO by adding a mask prediction branch that supports all image segmentation tasks
- The framework uses query embeddings from DINO to predict binary masks on a high-resolution pixel embedding map
- Key components in DINO are extended for segmentation through a shared architecture and training process
- Mask DINO is simple, efficient, scalable, and benefits from joint large-scale detection and segmentation datasets
- In experiments, Mask DINO significantly outperforms all existing specialized segmentation methods on both ResNet-50 backbone and pre-trained model with SwinL backbone
- Mask DINO establishes the best results to date on instance segmentation (54.5 AP on COCO), panoptic segmentation (59.4 PQ on COCO), and semantic segmentation (60.8 mIoU on ADE20K)
- The authors plan to make the code available at https://github.com/IDEACVR/MaskDINO
- Overall, Mask DINO presents an innovative approach to unifying object detection and segmentation using transformer-based models that achieve state-of-the-art performance across multiple tasks.

## Summary

Mask DINO is a transformer-based framework for object detection and segmentation that extends DINO (DETR with Improved Denoising Anchor Boxes) by adding a mask prediction branch that supports all image segmentation tasks, including instance, panoptic, and semantic. The framework uses the query embeddings from DINO to dot-product a high-resolution pixel embedding map to predict a set of binary masks. Key components in DINO are extended for segmentation through a shared architecture and training process. The authors note that Mask DINO is simple, efficient, scalable, and benefits from joint large-scale detection and segmentation datasets. In their experiments, they found that Mask DINO significantly outperforms all existing specialized segmentation methods on both ResNet-50 backbone and pre-trained model with SwinL backbone. Notably, Mask DINO establishes the best results to date on instance segmentation (54.5 AP on COCO), panoptic segmentation (59.4 PQ on COCO), and semantic segmentation (60.8 mIoU on ADE20K). The authors plan to make the code available at https://github.com/IDEACVR/MaskDINO for further use by researchers in this field. Overall, Mask DINO presents an innovative approach to unifying object detection and segmentation using transformer-based models that achieve state-of-the-art performance across multiple tasks.