# Class-agnostic Object Detection with Multi-modal Transformer

**Authors:** Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, Ming-Hsuan Yang  
**arXiv:** [2111.11430](https://arxiv.org/abs/2111.11430)  
**Date:** 2021-11-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2111.11430v6/

## Key points

- Existing methods lack a top-down supervision signal based on human-understandable semantics
- Multi-modal Vision Transformers (MViTs) trained with aligned image-text pairs effectively bridge this gap
- MViTs showcase state-of-the-art performance in localizing generic objects in images
- Existing MViTs lack multi-scale feature processing and require longer training schedules
- Proposed efficient MViT architecture using multi-scale deformable attention and late vision-language fusion
- MViTs have applications in open-world object detection, salient and camouflage object detection, as well as supervised and self-supervised detection tasks
- MViTs offer enhanced interactability by adaptively generating proposals based on specific language queries
- Detailed explanations of existing architectures such as GPV-I and MDETR, highlighting their strengths and weaknesses
- Proposed MAVL architecture combines multi-scale image features with multi-scale deformable attention modules for improved performance
- Language structure available in image-caption pairs used for training contributes to the improved performance of MViTs
- Impressive performance of MViTs in generic object detection across various domains showcased
- More flexible and efficient MViT architecture developed for off-the-shelf class-agnostic object detection, customizable with different text queries to generate desired proposal sets

## Summary

, , , , 

In the field of computer vision, the question of what constitutes an object has long been a topic of debate. Various approaches have been developed to score objectness, but they often struggle to scale well across different domains and novel objects. This paper argues that existing methods lack a top-down supervision signal based on human-understandable semantics. For the first time, the authors demonstrate that Multi-modal Vision Transformers (MViTs) trained with aligned image-text pairs can effectively bridge this gap. The authors conduct extensive experiments across various domains and novel objects to showcase the state-of-the-art performance of MViTs in localizing generic objects in images. They also identify that existing MViTs lack multi-scale feature processing and require longer training schedules. To address these limitations, the authors propose an efficient MViT architecture using multi-scale deformable attention and late vision-language fusion. The significance of MViT proposals is demonstrated in a diverse range of applications, including open-world object detection, salient and camouflage object detection, as well as supervised and self-supervised detection tasks. Additionally, MViTs offer enhanced interactability by adaptively generating proposals based on specific language queries. The paper provides detailed explanations of existing architectures such as GPV-I and MDETR, highlighting their strengths and weaknesses. The proposed MAVL architecture combines multi-scale image features with multi-scale deformable attention modules for improved performance. Through systematic experiments, the authors analyze the factors contributing to the improved performance of MViTs. They emphasize the role of language structure available in image-caption pairs used for training. In conclusion, this paper showcases the impressive performance of MViTs in generic object detection across various domains. The authors develop a more flexible and efficient MViT architecture for off-the-shelf class-agnostic object detection, which can be customized with different text queries to generate desired proposal sets. The use-cases for class-agnostic proposals are explored extensively, demonstrating their potential in improving performance in different scenarios.