PointCLIP: Point Cloud Understanding by CLIP

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Cites this paper Not relevant

   PointCLIP V2: Adapting CLIP for Powerful 3D Open-world Learning

   Xiangyang Zhu, Renrui Zhang, Bowei He et al. · 2022 · Computer Vision and Pattern Recognition · cited 303 times · AI summary available

   Abstract

   Contrastive Language-Image Pre-training (CLIP) has shown promising open-world performance on 2D image tasks, while its transferred capacity on 3D point clouds, i.e., PointCLIP, is still far from satisfactory. In this work, we propose PointCLIP V2, a powerful 3D open-world learner, to fully unleash the potential of CLIP on 3D point cloud data. First, we introduce a realistic shape projection module to generate more realistic depth maps for CLIP's visual encoder, which is quite efficient and narrows the domain gap between projected point clouds with natural images. Second, we leverage large-scale language models to automatically design a more descriptive 3D-semantic prompt for CLIP's textual e…

2. Very close Not relevant

   CLIP$^2$: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data

   Yihan Zeng, Chenhan Jiang, Jiageng Mao et al. · 2023 · Computer Vision and Pattern Recognition · cited 119 times · AI summary available

   Abstract

   Contrastive Language-Image Pre-training, benefiting from large-scale unlabeled text-image pairs, has demonstrated great performance in open-world vision understanding tasks. However, due to the limited Text-3D data pairs, adapting the success of 2D Vision-Language Models (VLM) to the 3D space remains an open problem. Existing works that leverage VLM for 3D understanding generally resort to constructing intermediate 2D representations for the 3D data, but at the cost of losing 3D geometry information. To take a step toward open-world 3D vision understanding, we propose Contrastive Language-Image-Point Cloud Pretraining (CLIP$^2$) to directly learn the transferable 3D point cloud representatio…

3. Very close Cites this paper Not relevant

   CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP

   Runnan Chen, Youquan Liu, Lingdong Kong et al. · 2023 · Computer Vision and Pattern Recognition · cited 286 times · AI summary available

   Abstract

   Contrastive Language-Image Pre-training (CLIP) achieves promising results in 2D zero-shot and few-shot learning. Despite the impressive performance in 2D, applying CLIP to help the learning in 3D scene understanding has yet to be explored. In this paper, we make the first attempt to investigate how CLIP knowledge benefits 3D scene understanding. We propose CLIP2Scene, a simple yet effective framework that transfers CLIP knowledge from 2D image-text pre-trained models to a 3D point cloud network. We show that the pre-trained 3D network yields impressive performance on various downstream tasks, i.e., annotation-free and fine-tuning with labelled data for semantic segmentation. Specifically, bu…

4. Very close Cites this paper Not relevant

   CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-Training

   Tianyu Huang, Bo-Wen Dong, Yun-Han Yang et al. · 2022 · IEEE International Conference on Computer Vision · cited 263 times · Not summarized yet

   Abstract

   Pre-training across 3D vision and language remains under development because of limited training data. Recent works attempt to transfer vision-language (V-L) pre-training methods to 3D vision. However, the domain gap between 3D and images is unsolved, so that V-L pre-trained models are restricted in 3D downstream tasks. To address this issue, we propose CLIP2Point, an image-depth pre-training method by contrastive learning to transfer CLIP to the 3D domain, and adapt it to point cloud classification. We introduce a new depth rendering setting that forms a better visual effect, and then render 52,460 pairs of images and depth maps from ShapeNet for pre-training. The pre-training scheme of CLI…

5. Very close Not relevant

   CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition

   Deepti Hegde, Jeya Maria Jose Valanarasu, Vishal M. Patel · 2023 · 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · cited 115 times · Not summarized yet

   Abstract

   Vision-Language models like CLIP have been widely adopted for various tasks due to their impressive zero-shot capabilities. However, CLIP is not suitable for extracting 3D geometric features as it was trained on only images and text by natural language supervision. We work on addressing this limitation and propose a new framework termed CG3D (CLIP Goes 3D) where a 3D encoder is learned to exhibit zero-shot capabilities. CG3D is trained using triplets of pointclouds, corresponding rendered 2D images, and texts using natural language supervision. To align the features in a multimodal embedding space, we utilize contrastive loss on 3D features obtained from the 3D encoder, as well as visual and…

6. Very close Not relevant

   DiffCLIP: Leveraging Stable Diffusion for Language Grounded 3D Classification

   Si-Tian Shen, Zi-Lin Zhu, Lin-Qian Fan et al. · 2023 · IEEE Workshop/Winter Conference on Applications of Computer Vision · cited 60 times · Not summarized yet

   Abstract

   Large pre-trained models have revolutionized the field of computer vision by facilitating multi-modal learning. Notably, the CLIP model has exhibited remarkable proficiency in tasks such as image classification, object detection, and semantic segmentation. Nevertheless, its efficacy in processing 3D point clouds is restricted by the domain gap between the depth maps derived from 3D projection and the training images of CLIP.This paper introduces DiffCLIP, a novel pre-training framework that seamlessly integrates stable diffusion with ControlNet. The primary objective of DiffCLIP is to bridge the domain gap inherent in the visual branch. Furthermore, to address few-shot tasks in the textual b…

7. Very close Cites this paper Uses this paper in depth Not relevant

   CLIPoint3D: Language-Grounded Few-Shot Unsupervised 3D Point Cloud Domain Adaptation

   Mainak Singha, Sarthak Mehrotra, Paolo Casari et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet

   Abstract

   Recent vision-language models (VLMs) such as CLIP demonstrate impressive cross-modal reasoning, extending beyond images to 3D perception. Yet, these models remain fragile under domain shifts, especially when adapting from synthetic to real-world point clouds. Conventional 3D domain adaptation approaches rely on heavy trainable encoders, yielding strong accuracy but at the cost of efficiency. We introduce CLIPoint3D, the first framework for few-shot unsupervised 3D point cloud domain adaptation built upon CLIP. Our approach projects 3D samples into multiple depth maps and exploits the frozen CLIP backbone, refined through a knowledge-driven prompt tuning scheme that integrates high-level lang…

8. Very close Cites this paper Not relevant

   Parameter-Efficient CLIP Adaptation for 3D Understanding via Unified Tokenization

   Guo-Feng Mei, Qin-Feng Xiao, Bin Ren et al. · 2025 · cited 1 time · Not summarized yet

   Abstract

   Vision-language models, such as CLIP, encode rich semantic knowledge through large-scale image-text pretraining. Reusing these models for 3D understanding is highly desirable, because 3D-text pairs and dense point-level annotations are far scarcer and more difficult to obtain than their 2D counterparts. However, CLIP is trained on regular 2D image patches, whereas point clouds are unordered, sparse, and irregular. The primary challenge lies in constructing 3D token sequences whose geometry, position, and local relations can be interpreted by a frozen, pretrained vision transformer. Moreover, coordinate-scale variations across heterogeneous 3D domains make unified training particularly challe…

Prior work it builds on

1. Close Used in depth by this paper A method it uses Not relevant

   PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

   R. Charles, Hao Su, K. Mo et al. · 2016 · Computer Vision and Pattern Recognition · cited 19,029 times · Not summarized yet

   Abstract

   Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better th…

2. Close Used in depth by this paper A method it uses Not relevant

   Learning Transferable Visual Models From Natural Language Supervision

   Alec Radford, Jong Wook Kim, Chris Hallacy et al. · 2021 · Computer Vision and Pattern Recognition · cited 59,490 times · AI summary available

   Abstract

   State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to refer…

3. Close Used in depth by this paper Not relevant

   Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

   Renrui Zhang, Rongyao Fang, Peng Gao et al. · 2021 · Computer Vision and Pattern Recognition · preprint, not peer reviewed · cited 624 times · AI summary available

   Abstract

   Contrastive Vision-Language Pre-training, known as CLIP, has provided a new paradigm for learning visual representations by using large-scale contrastive image-text pairs. It shows impressive performance on zero-shot knowledge transfer to downstream tasks. To further enhance CLIP's few-shot capability, CLIP-Adapter proposed to fine-tune a lightweight residual feature adapter and significantly improves the performance for few-shot classification. However, such a process still needs extra training and computational resources. In this paper, we propose \textbfTraining-Free CL\textbf{IP}-\textbf{Adapter} (\textbf{Tip-Adapter}), which not only inherits CLIP's training-free advantage but also perf…

4. Close* Not relevant

   Image2Point: 3D Point-Cloud Understanding with Pretrained 2D ConvNets

   Chen-Feng Xu, Shijia Yang, Bohan Zhai et al. · 2021 · arXiv.org · preprint, not peer reviewed · cited 26 times · Not summarized yet
5. Close Used in depth by this paper A method it uses Not relevant

   Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline

   Ankit Goyal, Hei Law, Bowei Liu et al. · 2021 · International Conference on Machine Learning · cited 317 times · Not summarized yet

   Abstract

   Processing point cloud data is an important component of many real-world systems. As such, a wide variety of point-based approaches have been proposed, reporting steady benchmark improvements over time. We study the key ingredients of this progress and uncover two critical results. First, we find that auxiliary factors like different evaluation schemes, data augmentation strategies, and loss functions, which are independent of the model architecture, make a large difference in performance. The differences are large enough that they obscure the effect of architecture. When these factors are controlled for, PointNet++, a relatively older network, performs competitively with recent methods. Sec…

Papers that cite it

1. Very close Not relevant

   ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding

   Le Xue, Mingfei Gao, Chen Xing et al. · 2022 · Computer Vision and Pattern Recognition · cited 455 times · Not summarized yet

   Abstract

   The recognition capabilities of current state-of-the-art 3D models are limited by datasets with a small number of annotated data and a pre-defined set of categories. In its 2D counterpart, recent advances have shown that similar problems can be significantly alleviated by employing knowledge from other modalities, such as language. Inspired by this, leveraging multimodal information for 3D modality could be promising to improve 3D understanding under the restricted data regime, but this line of research is not well studied. Therefore, we introduce ULIP to learn a unified representation of image, text, and 3D point cloud by pre-training with object triplets from the three modalities. To overc…

2. Very close* Not relevant

   MVF-PointCLIP: Training-free multi-view fusion PointCLIP for zero-shot 3D classification

   Jiu-Qian Dai, Zhenyan Ji, Ze-Chang Xiong et al. · 2025 · Neurocomputing · cited 7 times · Not summarized yet
3. Close Uses this paper in depth Not relevant

   FILP-3D: Enhancing 3D few-shot class-incremental learning with pre-trained vision-language models

   Wan Xu, Tian-Yu Huang, Tian-Yu Qu et al. · 2025 · Pattern Recognition · cited 12 times · Not summarized yet

   Abstract

   Few-shot class-incremental learning (FSCIL) aims to mitigate the catastrophic forgetting issue when a model is incrementally trained on limited data. While the Con-trastive Vision-Language Pre-Training (CLIP) model has been effective in addressing 2D few/zero-shot learning tasks, its direct application to 3D FSCIL faces limitations. These limitations arise from feature space misalignment and significant noise in real-world scanned 3D data. To address these challenges, we introduce two novel components: the Redundant Feature Eliminator (RFE) and the Spatial Noise Compensator (SNC). RFE aligns the feature spaces of input point clouds and their embeddings by performing a unique dimensionality r…

4. Close Not relevant

   TriCLIP-3D: A Unified Parameter-Efficient Framework for Tri-Modal 3D Visual Grounding based on CLIP

   Fan Li, Zan-Yi Wang, Ze-Yi Huang et al. · 2025 · ACM Multimedia · cited 3 times · Not summarized yet

   Abstract

   3D visual grounding allows an embodied agent to understand visual information in real-world 3D environments based on human instructions, which is crucial for embodied intelligence. Existing 3D visual grounding methods typically rely on separate encoders for different modalities (e.g., RGB images, text, and 3D point clouds), resulting in large and complex models that are inefficient to train. While some approaches use pre-trained 2D multi-modal models like CLIP for 3D tasks, they still struggle with aligning point cloud data to 2D encoders. As a result, these methods continue to depend on 3D encoders for feature extraction, further increasing model complexity and training inefficiency. In thi…

5. Close Not relevant

   SmartCLIP: Modular Vision-language Alignment with Identification Guarantees

   Shao-An Xie, Ling-Jing Kong, Yu-Jian Zheng et al. · 2025 · Computer Vision and Pattern Recognition · cited 24 times · Not summarized yet

   Abstract

   Contrastive Language-Image Pre-training (CLIP) [37] has emerged as a pivotal model in computer vision and multimodal learning, achieving state-of-the-art performance at aligning visual and textual representations through contrastive learning. However, CLIP struggles with potential information misalignment in many image-text datasets and suffers from entangled representation. On the one hand, short captions for a single image in datasets like MSCOCO may describe disjoint regions in the image, leaving the model uncertain about which visual features to retain or disregard. On the other hand, directly aligning long captions with images can lead to the retention of entangled details, preventing t…