# CLIP$^2$: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data

**Authors:** Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung, Zhen Yang, Xiaodan Liang, Hang Xu  
**arXiv:** [2303.12417](https://arxiv.org/abs/2303.12417)  
**Date:** 2023-03-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.12417v2/

## Key points

- Contrastive Language-Image Pre-training has shown impressive performance in open-world vision understanding tasks
- Applying this success to 3D space remains a challenge due to limited availability of Text-3D data pairs
- Existing approaches for 3D understanding often rely on constructing intermediate 2D representations, resulting in loss of 3D geometry information
- Proposed solution: Contrastive Language-Image-Point Cloud Pretraining (CLIP$^2$) directly learns transferable 3D point cloud representations using a novel proxy alignment mechanism
- Approach exploits naturally existing correspondences between 2D and 3D scenarios and builds well-aligned and instance-based text-image-point proxies from complex scenes
- Cross-modal contrastive objective is introduced to learn semantic and instance-level aligned point cloud representations
- Experimental results demonstrate strong transferability of learned 3D representation in downstream tasks, outperforming state-of-the-art methods
- Capabilities of different representations in real world scenarios are analyzed, optional ensemble scheme presented to improve performance
- Research contributes to advancing open world 3D vision understanding by learning transferable 3D point cloud representations without losing important geometry information

## Summary

Contrastive Language-Image Pre-training has shown impressive performance in open-world vision understanding tasks by leveraging large-scale unlabeled text-image pairs. However, applying this success to 3D space remains a challenge due to the limited availability of Text-3D data pairs. Existing approaches that use Vision-Language Models (VLM) for 3D understanding often rely on constructing intermediate 2D representations, which results in the loss of 3D geometry information. To address this issue, we propose Contrastive Language-Image-Point Cloud Pretraining (CLIP$^2$), which directly learns transferable 3D point cloud representations in realistic scenarios using a novel proxy alignment mechanism. Our approach exploits naturally existing correspondences between 2D and 3D scenarios and builds well-aligned and instance-based text-image-point proxies from complex scenes. We also introduce a cross-modal contrastive objective to learn semantic and instance-level aligned point cloud representations. Experimental results on both indoor and outdoor scenarios demonstrate that our learned 3D representation exhibits strong transferability in downstream tasks, including zero-shot and few-shot 3D recognition, significantly outperforming state-of-the art methods. Furthermore, we analyze the capabilities of different representations in real world scenarios and present an optional ensemble scheme to further improve performance. Our work contributes to advancing open world 3D vision understanding by directly learning transferable 3D point cloud representations without losing important geometry information. This research will be presented at CVPR 2023 conference.