# M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts

**Authors:** Mingsheng Li, Xin Chen, Chi Zhang, Sijin Chen, Hongyuan Zhu, Fukun Yin, Gang Yu, Tao Chen  
**arXiv:** [2312.10763](https://arxiv.org/abs/2312.10763)  
**Date:** 2023-12-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.10763v1/

## Key points

- Introduction of M3DBench, a comprehensive 3D instruction-following dataset
- Importance of 3D understanding in autonomous agents for decision-making
- Limitations of existing datasets and methods that are task-specific
- Motivation to explore MLMs' potential for 3D tasks
- Lack of large-scale 3D instruction-following datasets
- M3DBench as a solution with support for general multimodal instructions, unification of diverse 3D tasks, and large-scale size (over 320k instruction-response pairs)
- Establishment of a new benchmark for assessing performance of large models in understanding multi-modal 3D prompts
- Extensive experiments conducted using M3DBench and baseline model to demonstrate effectiveness in supporting general 3D-centric tasks
- Overall contribution of the paper in providing a comprehensive dataset and benchmark for future research in leveraging MLMs for broader applications in the field.

## Summary

The paper titled "M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts" introduces a comprehensive 3D instruction-following dataset called M3DBench. The authors highlight the importance of 3D understanding in facilitating autonomous agents for decision-making and identify the limitations of existing datasets and methods that are often task-specific. This motivates the exploration of MLMs' potential to be 3D generalists for a wider range of tasks. However, current research on MLMs has been less focused on 3D tasks due to the lack of large-scale 3D instruction-following datasets. To address this gap, the authors present M3DBench as a solution. M3DBench is a valuable resource for training and evaluating large models due to its support for general multimodal instructions, unification of diverse 3D tasks at both region and scene levels, and its large-scale size with over 320k instruction-response pairs. In addition to introducing M3DBench, the authors establish a new benchmark for assessing the performance of large models in understanding multi-modal 3D prompts. They conduct extensive experiments using their dataset and baseline model to demonstrate its effectiveness in supporting general 3D-centric tasks. Overall, this paper presents an important contribution by providing a comprehensive  and benchmark that can inspire future research in leveraging  for broader applications in the field of , , , and decision-making.