# What is the Visual Cognition Gap between Humans and Multimodal LLMs?

**Authors:** Xu Cao, Bolin Lai, Wenqian Ye, Yunsheng Ma, Joerg Heintz, Jintai Chen, Jianguo Cao, James M. Rehg  
**arXiv:** [2406.10424](https://arxiv.org/abs/2406.10424)  
**Date:** 2024-06-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.10424v1/

## Key points

- Study focused on exploring the gap between humans and machine learning language models (MLLMs)
- MLLMs show promise in recognition and object detection but effectiveness in high-level reasoning tasks is uncertain
- Researchers highlighted Audio-Visual Reasoning (AVR) as a significant challenge for MLLMs
- Proposed new dataset called MaRs-VQA and benchmark named VCog-Bench to evaluate zero-shot AVR capability of MLLMs
- Comparative experiments on VCog-Bench revealed a gap between MLLMs and human intelligence in visual cognitive abilities
- Release of VCog-Bench and MaRs-VQA dataset expected to drive progress towards developing next-generation MLLMs with human-like visual cognition abilities
- Study emphasizes the importance of further research to bridge the gap between current MLLMs and human-level visual reasoning capabilities

## Summary

In a recent study by Xu Cao, Bolin Lai, Wenqian Ye, Yunsheng Ma, Joerg Heintz, Jintai Chen, Jianguo Cao, and James M. Rehg from various academic institutions including the University of Illinois Urbana-Champaign and Georgia Institute of Technology, the focus was on exploring the  between humans and . These MLLMs have shown promise in tasks like recognition and object detection but their effectiveness in high-level reasoning tasks remains uncertain. The researchers highlighted  as a significant challenge for MLLMs. AVR involves discerning relationships among patterns in images to predict subsequent patterns. This cognitive ability is crucial during early neurodevelopmental stages in children. Inspired by tasks like Raven's Progressive Matrices (RPM) and Wechsler Intelligence Scale for Children (WISC), the team proposed a new dataset called  and a benchmark named  containing three datasets to evaluate zero-shot AVR capability of MLLMs. Through comparative experiments with different open-source and closed-source MLLMs on VCog-Bench, the researchers discovered a gap between MLLMs and human intelligence in terms of visual cognitive abilities. This highlights the limitations of current MLLMs in handling complex visual reasoning tasks. The release of VCog-Bench along with MaRs-VQA dataset and inference pipeline is expected to drive progress towards developing next-generation MLLMs with human-like visual cognition abilities. The study sheds light on the challenges faced by MLLMs in achieving human-level visual reasoning capabilities and emphasizes the importance of further research to bridge this gap. The findings provide valuable insights for advancing artificial intelligence systems towards more sophisticated cognitive abilities akin to those exhibited by humans.