# Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively

**Authors:** Haobo Yuan, Xiangtai Li, Chong Zhou, Yining Li, Kai Chen, Chen Change Loy  
**arXiv:** [2401.02955](https://arxiv.org/abs/2401.02955)  
**Date:** 2024-01-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.02955v1/

## Key points

- CLIP and SAM are highly regarded vision foundation models (VFMs)
- Open-Vocabulary SAM is a novel framework that integrates CLIP and SAM
- Open-Vocabulary SAM leverages knowledge transfer modules: SAM2CLIP and CLIP2SAM
- Extensive experiments were conducted to evaluate the effectiveness of Open-Vocabulary SAM
- Open-Vocabulary SAM outperforms naive baselines that combine SAM and CLIP
- Integration of CLIP and SAM significantly improves segmentation and recognition performance
- Image classification data training extends the capabilities of Open-Vocabulary SAM to recognize 22,000 classes
- The proposed framework demonstrates scalability and versatility
- Experimental results show state-of-the-art performance in both tasks with Open-Vocabulary SAM.

## Summary

, , , , 

The CLIP and Segment Anything Model (SAM) are two highly regarded vision foundation models (VFMs) known for their exceptional performance in segmentation and recognition tasks, respectively. In this paper, the authors propose a novel framework called Open-Vocabulary SAM that integrates these two models to achieve simultaneous interactive segmentation and recognition. The Open-Vocabulary SAM model leverages two unique knowledge transfer modules: SAM2CLIP and CLIP2SAM. The SAM2CLIP module adapts the knowledge from SAM into CLIP using distillation and learnable transformer adapters, allowing CLIP to benefit from SAM's expertise in segmentation tasks across diverse domains. On the other hand, the CLIP2SAM module transfers the recognition capabilities of CLIP into SAM, enhancing its ability to recognize objects. To evaluate the effectiveness of Open-Vocabulary SAM, extensive experiments were conducted on various datasets and detectors. The results demonstrate that Open-Vocabulary SAM outperforms naive baselines that simply combine SAM and CLIP. The integration of these two models significantly improves both segmentation and recognition performance. Furthermore, by incorporating image classification data training, the authors were able to extend the capabilities of Open-Vocabulary SAM to segment and recognize approximately 22,000 classes. This demonstrates the scalability and versatility of the proposed framework. Overall, this paper presents a comprehensive exploration of integrating the CLIP and SAM models into a unified framework for simultaneous interactive segmentation and recognition. The experimental results highlight the effectiveness of Open-Vocabulary SAM in achieving state-of-the-art performance in both tasks, surpassing simple combinations of individual models.