# Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond

**Authors:** Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Tianyu Liu, Baobao Chang  
**arXiv:** [2310.02071](https://arxiv.org/abs/2310.02071)  
**Date:** 2023-10-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.02071v1/

## Key points

- Researchers explore the potential of Multimodal Large Language Models (MLLMs) in improving embodied decision-making processes for agents.
- Introduction of a new benchmark called PCA-EVAL, which evaluates embodied decision-making from the perspectives of Perception, Cognition, and Action.
- Annotation process consists of Dataset Annotation and Dataset Refinement stages.
- Evaluation process for assessing state-of-the-art Visual Large Language Models (VLLMs) on end-to-end embodied decision-making using PCA-EVAL is detailed.
- Proposal of HOLMES, a multi-agent cooperation framework that allows collaboration between LLMs and MLLMs to enhance decision-making.
- Comparison between end-to-end embodied decision-making with HOLMES shows GPT4-Vision model outperforming GPT4-HOLMES in terms of average decision accuracy (+3%).
- GPT4-Vision surpasses open-source state-of-the-art MLLM by 26% in terms of performance.
- Powerful MLLMs like GPT4-Vision hold promise for decision-making in embodied agents.
- Overall, the study highlights the potential of MLLMs in improving embodied decision-making processes and introduces a new benchmark for evaluation.

## Summary

, , , , 

In this study, the researchers explore the potential of Multimodal Large Language Models (MLLMs) in improving embodied decision-making processes for agents. They introduce a new benchmark called PCA-EVAL, which evaluates embodied decision-making from the perspectives of Perception, Cognition, and Action. The annotation process consists of two stages: Dataset Annotation and Dataset Refinement. During the initial stage, annotators pinpoint informative source images and write questions for each image. In the subsequent stage, annotators scrutinize the output actions and rationales presented by ChatGPT to refine annotations and ensure a single correct answer. The evaluation process for assessing state-of-the-art Visual Large Language Models (VLLMs) on end-to-end embodied decision-making using PCA-EVAL is detailed. End-to-end decision making involves directly feeding visual observations and textual questions to the multi-modal agent. The agent outputs image descriptions and reasoning processes before giving the final action. Additionally, the researchers propose HOLMES, a multi-agent cooperation framework that allows large language models like ChatGPT-3.5 and GPT4 to call different visual models or APIs to gather information about the environment. HOLMES enables collaboration between LLMs and MLLMs to enhance decision-making. The researchers compare end-to-end embodied decision-making with HOLMES on their benchmark and find that the GPT4-Vision model demonstrates strong end-to-end embodied decision-making abilities, outperforming GPT4-HOLMES in terms of average decision accuracy (+3%). However, this performance is exclusive to GPT4-Vision, surpassing the open-source state-of-the-art MLLM by 26%. These results suggest that powerful MLLMs like GPT4-Vision hold promise for decision-making in embodied agents. Overall, this study highlights the potential of MLLMs in improving embodied decision-making processes for agents and introduces a new benchmark for evaluating such capabilities. The proposed HOLMES framework allows for collaboration between LLMs and MLLMs, further enhancing decision-making abilities. The findings contribute to the advancement of MLLM research and offer new avenues for improving decision-making in embodied agents.