# ShapeLLM: Universal 3D Object Understanding for Embodied Interaction

**Authors:** Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, Kaisheng Ma  
**arXiv:** [2402.17766](https://arxiv.org/abs/2402.17766)  
**Date:** 2024-02-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.17766v1/

## Key points

- ShapeLLM is introduced as the first 3D Multimodal Large Language Model (LLM) tailored for embodied interaction.
- The model integrates 3D point clouds and natural language processing for a comprehensive understanding of 3D objects.
- ShapeLLM builds upon an enhanced 3D encoder known as [name not provided], which leverages multi-view image distillation to enhance geometric comprehension.
- The model undergoes training on instruction-following datasets and is evaluated on a benchmark named [name not provided].
- Results show that ShapeLLM excels in achieving state-of-the-art performance in tasks related to 3D geometry understanding and language-driven interactions within interactive environments.
- This advancement signifies progress in bridging the gap between 3D object perception and linguistic communication in interactive settings.

## Summary

In their paper titled "ShapeLLM: Universal 3D Object Understanding for Embodied Interaction," authors Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma introduce ShapeLLM as the first 3D Multimodal Large Language Model (LLM) tailored for embodied interaction. This innovative model delves into a comprehensive understanding of 3D objects through the integration of 3D point clouds and natural language processing. builds upon an enhanced 3D encoder known as , an extension of ReCon that leverages multi-view image distillation to bolster geometric comprehension. By employing  as the input encoder for LLMs,  undergoes training on meticulously crafted instruction-following datasets and is subsequently evaluated on a newly curated benchmark named . The results showcase that both  and  excel in achieving state-of-the-art performance in tasks related to 3D geometry understanding and language-driven interactions within the realm of embodied visual grounding. This signifies a significant advancement in bridging the gap between 3D object perception and linguistic communication within interactive environments.