# MMToM-QA: Multimodal Theory of Mind Question Answering

**Authors:** Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua B. Tenenbaum, Tianmin Shu  
**arXiv:** [2401.08743](https://arxiv.org/abs/2401.08743)  
**Date:** 2024-01-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.08743v2/

## Key points

- Artificial intelligence plays a crucial role in developing machines with human-level social intelligence
- Recent advancements in machine learning, particularly large language models, show promise in understanding aspects of Theory of Mind (ToM)
- Existing ToM benchmarks primarily rely on unimodal datasets, which do not fully capture the complexity of human ToM reasoning
- A new benchmark called [Benchmark Name] has been introduced to evaluate machine ToM on both multimodal and unimodal data related to a person's activities in a household environment
- A novel method known as [Novel Method Name] has been proposed to enhance multimodal ToM capacity by extracting unified representations from multimodal data and leveraging language models for scalable Bayesian inverse planning
- BIP-ALM showed promising results by combining model-based mental inference with language models effectively, outperforming state-of-the-art models like GPT-4
- The evaluation protocol involved testing the models under three conditions: Multimodal QA, Text QA, and Video QA in zero-shot evaluation setting
- BIP-ALM utilizes Bayesian Inverse Planning to infer a person's mental state based on video and text inputs, extending traditional Bayesian inverse planning methods.

## Summary

In the realm of artificial intelligence,  plays a crucial role in developing machines with human-level social intelligence. Recent advancements in machine learning, particularly through large language models, have shown some promise in understanding aspects of ToM. However, existing ToM benchmarks primarily rely on unimodal datasets, such as video or text, which do not fully capture the complexity of human ToM reasoning. To address this limitation, a new benchmark called  has been introduced. This benchmark evaluates machine ToM on both multimodal and unimodal data related to a person's activities in a household environment. To enhance multimodal ToM capacity, a novel method known as  has been proposed. BIP-ALM extracts unified representations from multimodal data and leverages language models for scalable Bayesian inverse planning. A systematic comparison was conducted between human performance, BIP-ALM, and state-of-the-art models like GPT-4. The results revealed that while large language models and multimodal models still struggle with robust ToM capacity, BIP-ALM showed promising results by combining model-based mental inference with language models effectively. The evaluation protocol involved testing the models under three conditions: Multimodal QA with both video and text inputs present, Text QA with only text input, and Video QA with only video input. The zero-shot evaluation setting required models to generalize learned knowledge from training data without specific examples provided during training. The BIP-ALM model utilizes Bayesian Inverse Planning to infer a person's mental state based on video and text inputs. By building unified representations from multimodal inputs and fine-tuning language models for efficient inverse symbolic planning, BIP-ALM extends the capabilities of traditional Bayesian inverse planning methods. Overall, the introduction of  along with the innovative  represents significant contributions towards advancing machine understanding of Theory of Mind in complex real-world scenarios.