# Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Driving

**Authors:** Mahyar Najibi, Jingwei Ji, Yin Zhou, Charles R. Qi, Xinchen Yan, Scott Ettinger, Dragomir Anguelov  
**arXiv:** [2309.14491](https://arxiv.org/abs/2309.14491)  
**Date:** 2023-09-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.14491v1/

## Key points

- Accurate and reliable models are crucial in the field of autonomous driving
- Existing models may not be sufficient for safety critical applications as they are trained on a pre-defined set of object categories
- Proposed method can generate amodal 3D bounding boxes and tracklets for training models on open-set categories without 3D human labels
- Pipeline leverages motion cues in point cloud sequences and 2D image-text pairs to identify and track traffic participants
- Proposed method can handle both static and moving objects in an unsupervised manner
- Capable of outputting open-vocabulary features through vision-language knowledge distillation technique
- Experiments conducted on Waymo Open Dataset show superior performance compared to prior work on unsupervised 3D perception tasks
- Pipeline involves extracting vision-language and motion features, proposing, tracking, and completing bounding boxes, utilizing all available cameras, transferring 2D features to 3D LiDAR points, incorporating point cloud data captured over time using LiDAR sensors
- Novel approach for unsupervised multi-modal auto labeling in autonomous driving

## Summary

In the field of , it is crucial to have accurate and reliable . However, these models may not be sufficient for safety critical applications as they are trained on a pre-defined set of object categories. This poses a problem when new object types are encountered after deployment, as these models may not be able to accurately detect and track them. To address this issue, the authors propose a  that can generate amodal 3D bounding boxes and tracklets for training models on open-set categories without the need for 3D human labels. The pipeline leverages motion cues inherent in point cloud sequences along with freely available 2D image-text pairs to identify and track all traffic participants. Unlike previous studies in this domain that can only provide class-agnostic auto labels limited to moving objects, the proposed method can handle both static and moving objects in an unsupervised manner. Additionally, it is capable of outputting  thanks to the vision-language knowledge distillation technique. The authors conduct experiments on the Waymo Open Dataset to evaluate their approach. The results show that their method outperforms prior work by significant margins on various unsupervised 3D perception tasks. Figure 3 provides an overview of the unsupervised multi-modal auto labeling approach. The pipeline begins by extracting vision-language and motion features from multiple modalities. These features are then used to propose, track, and complete bounding boxes of objects. The resulting pointwise vision-language features, 3D bounding boxes, and tracklets serve as automatic supervisions to train the perception model. To extract open-vocabulary features, the authors utilize all available cameras to capture images at each time step. These 2D features are then transferred to 3D LiDAR points using known sensor calibrations. The point cloud data captured over time using LiDAR sensors is also incorporated into the feature extraction process. Overall, this paper presents a novel approach for  in autonomous driving. By combining motion cues and vision-language knowledge distillation, the proposed method can effectively handle both static and moving objects without the need for human-labeled 3D data. The experimental results demonstrate the superiority of this approach compared to previous work in various unsupervised 3D perception tasks.