# PointCLIP V2: Adapting CLIP for Powerful 3D Open-world Learning

**Authors:** Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyao Zeng, Shanghang Zhang, Peng Gao  
**arXiv:** [2211.11682](https://arxiv.org/abs/2211.11682)  
**Date:** 2022-11-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2211.11682v1/

## Key points

- CLIP has been successful in 2D image tasks but not in 3D point clouds
- PointCLIP V2 is proposed to address this issue and unleash the potential of CLIP on 3D point cloud data
- PointCLIP V2 introduces a realistic shape projection module and leverages large-scale language models such as GPT-3 to automatically design a more descriptive 3D-semantic prompt for CLIP's textual encoder
- PointCLIP V2 significantly outperforms PointCLIP by +42.90%, +40.44%, and +28.75% accuracy on three datasets for zero-shot 3D classification without any training in 3D domains
- PointCLIP V2 can be extended to few-shot classification, zero-shot part segmentation, and zero-shot 3D object detection with superior generalization ability for 3D open-world learning
- The authors provide code at https://github.com/yangyangyang127/PointCLIP_V2 which includes comparisons between different textual encoders and their computation costs

## Summary

The Contrastive Language-Image Pre-training (CLIP) has been successful in open-world 2D image tasks, but its transferred capacity on 3D point clouds, known as PointCLIP, has not been satisfactory. To address this issue, the authors propose PointCLIP V2, a powerful 3D open-world learner that unleashes the potential of CLIP on 3D point cloud data. The approach introduces a realistic shape projection module to generate more realistic depth maps for CLIP's visual encoder and leverages large-scale language models such as GPT-3 to automatically design a more descriptive 3D-semantic prompt for CLIP's textual encoder. Without any training in 3D domains, PointCLIP V2 significantly outperforms PointCLIP by +42.90%, +40.44%, and +28.75% accuracy on three datasets for zero-shot 3D classification. Furthermore, PointCLIP V2 can be extended to few-shot classification, zero-shot part segmentation, and zero-shot 3D object detection with superior generalization ability for 3D open-world learning. The authors also provide code at https://github.com/yangyangyang127/PointCLIP_V2 which includes comparisons between different textual encoders and their computation costs. In addition to the existing summary, the new context provides further details about how GPT-3 is used to organize keywords into complete sentences and enrich additional shape related contents through textual prompts such as describing a depth map of a specific class or generating synonyms.