# CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP

**Authors:** Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, Wenping Wang  
**arXiv:** [2301.04926](https://arxiv.org/abs/2301.04926)  
**Date:** 2023-01-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2301.04926v2/

## Key points

- CLIP2Scene framework transfers CLIP knowledge from 2D image-text pre-trained models to a 3D point cloud network
- Semantic-driven Cross-modal Contrastive Learning framework used to train the 3D network with contrastive loss
- Consistency enforced between temporally coherent point cloud features and corresponding image features
- Experiments show effectiveness of CLIP2Scene in achieving annotation-free 3D semantic segmentation with high mIoU scores
- Generalizability demonstrated for handling cross-domain datasets
- Challenges faced by previous methods in unsupervised cross-modal knowledge distillation discussed and solutions proposed
- Dense pixel-text correspondence used for training sample selection
- Spatial-temporal consistency regularization introduced to consider temporal coherence of multi-sweep point clouds
- CLIP2Scene framework outperforms other self-supervised methods when fine-tuned with labeled data

## Summary

In this paper, the authors propose a framework called CLIP2Scene that transfers Contrastive Language-Image Pre-training (CLIP) knowledge from 2D image-text pre-trained models to a 3D point cloud network. The authors design a Semantic-driven Cross-modal Contrastive Learning framework to train the 3D network using contrastive loss based on positive and negative point samples selected by leveraging CLIP's text semantics. Additionally, they enforce consistency between temporally coherent point cloud features and their corresponding image features. The experiments conducted on SemanticKITTI, nuScenes, and ScanNet datasets demonstrate the effectiveness of CLIP2Scene in achieving annotation-free 3D semantic segmentation with significant mean Intersection over Union (mIoU) scores when fine-tuned with either 1% or 100% labeled data. Furthermore, the authors demonstrate the generalizability of their method for handling cross-domain datasets. They also discuss some challenges faced by previous methods in unsupervised cross-modal knowledge distillation and propose solutions such as using dense pixel-text correspondence for training sample selection and introducing spatial-temporal consistency regularization to consider temporal coherence of multi-sweep point clouds. In conclusion, this work explores how CLIP knowledge can benefit 3D scene understanding and presents the CLIP2Scene framework which outperforms other self-supervised methods when fine tuned with labeled data.