# COMMA: A Communicative Multimodal Multi-Agent Benchmark

**Authors:** Timothy Ossowski, Jixuan Chen, Danyal Maqbool, Zefan Cai, Tyler Bradshaw, Junjie Hu  
**arXiv:** [2410.07553](https://arxiv.org/abs/2410.07553)  
**Date:** 2024-10-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.07553v1/

## Key points

- Authors Timothy Ossowski, Jixuan Chen, Danyal Maqbool, Zefan Cai, Tyler Bradshaw, and Junjie Hu focus on multi-modal agents in language-based communication for collaborative tasks.
- They introduce a novel benchmark called COMMA to assess the performance of these agents in real-world deployments.
- The benchmark evaluates four key categories of agentic capability within a communicative collaboration setting.
- Testing both agent-agent and agent-human collaborations using open-source and closed-source models reveals weaknesses in state-of-the-art models like GPT-4o.
- Current multi-modal agents struggle to effectively communicate and collaborate with each other and with humans, indicating the need for further development in this area.

## Summary

In their paper titled "COMMA: A Communicative Multimodal Multi-Agent Benchmark," authors Timothy Ossowski, Jixuan Chen, Danyal Maqbool, Zefan Cai, Tyler Bradshaw, and Junjie Hu address the potential of multi-modal agents in language-based communication for collaborative tasks. They highlight a critical gap in understanding the effectiveness of these agents in real-world deployments and introduce a novel benchmark to assess their performance. The  encompasses a range of scenarios that evaluate four key categories of agentic capability within a communicative collaboration setting. By testing both agent-agent and agent-human collaborations using open-source and closed-source models, the study reveals surprising weaknesses in state-of-the-art models like GPT-4o. The findings indicate that current multi-modal agents struggle to effectively communicate and collaborate with each other and with humans. This emphasizes the need for further development and refinement in this area for successful real-world applications.