# PointCLIP: Point Cloud Understanding by CLIP

**Authors:** Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, Hongsheng Li  
**arXiv:** [2112.02413](https://arxiv.org/abs/2112.02413)  
**Date:** 2021-12-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2112.02413v1/

## Key points

- Significant progress in zero-shot and few-shot learning through Contrastive Vision-Language Pre-training (CLIP) for 2D visual recognition
- Uncertainty about whether CLIP can be generalized to 3D recognition
- Proposal of PointCLIP as a solution for aligning CLIP-encoded point clouds with 3D category texts
- Encoding of point cloud by projecting it into multi-view depth maps without rendering
- Aggregation of view-wise zero-shot predictions to transfer knowledge from 2D to 3D domain
- Design of an inter-view adapter to improve feature extraction and fuse few-shot knowledge from 3D data into CLIP pre-trained in 2D
- Fine tuning of lightweight adapter in few shot settings improves performance
- Complementary property observed between PointCLIP and classical 3D supervised networks
- Ensembling of models boosts baseline performance and surpasses state-of-the-art models
- PointCLIP offers effective understanding of 3D point clouds using CLIP under low resource cost and data regime

## Summary

Recently, there has been significant progress in zero-shot and few-shot learning through Contrastive Vision-Language Pre-training (CLIP) for 2D visual recognition. CLIP learns to match images with their corresponding texts in open-vocabulary settings. However, it remains unclear whether CLIP, which is pre-trained using large-scale image-text pairs in 2D, can be generalized to 3D recognition. In this paper, the authors propose PointCLIP as a feasible solution for aligning CLIP-encoded point clouds with 3D category texts. To achieve this alignment, the authors encode a point cloud by projecting it into multi-view depth maps without rendering. They then aggregate the view-wise zero-shot predictions to transfer knowledge from the 2D domain to the 3D domain. Additionally, they design an inter-view adapter that improves feature extraction and adaptively fuses few-shot knowledge learned from 3D data into CLIP pre-trained in 2D. By fine tuning the lightweight adapter in few shot settings, PointCLIP significantly improves performance. The authors also observe a complementary property between PointCLIP and classical 3D supervised networks. By ensembling these models together, PointCLIP surpasses state of the art models and boosts baseline performance. PointCLIP offers a promising alternative for effective understanding of 3D point clouds using CLIP under low resource cost and data regime. The authors conduct thorough experiments on widely adopted datasets such as ModelNet10, ModelNet40 and ScanObjectNN to demonstrate the effectiveness of PointCLIP.