# Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

**Authors:** Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, Xing Sun  
**arXiv:** [2405.21075](https://arxiv.org/abs/2405.21075)  
**Date:** 2024-05-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.21075v1/

## Key points

- Multi-modal Large Language Models (MLLMs) are a key focus in the pursuit of artificial general intelligence
- Progress has been made in static image understanding but a gap exists in processing sequential visual data
- Video-MME is the first-ever benchmark for assessing MLLMs in video analysis
- Video-MME offers diversity in video types, duration, breadth in data modalities, and quality annotations
- Encompasses six primary visual domains with 30 subfields to ensure broad scenario generalizability
- Dataset consists of 900 videos totaling 256 hours with 2,700 question-answer pairs for model evaluation
- Gemini 1.5 Pro identified as best-performing commercial model but need for enhancements in handling longer sequences and multi-modal data persists

## Summary

In the pursuit of artificial general intelligence, Multi-modal Large Language Models (MLLMs) have emerged as a key focus in recent advancements. Significant progress has been made in enhancing their capabilities in static image understanding. However, there remains a notable gap in exploring their potential for processing sequential visual data. This deficiency underscores the need for a comprehensive and high-quality evaluation of MLLMs' performance in this domain. To address this gap, we present Video-MME, the first-ever full-spectrum Multi-Modal Evaluation benchmark designed specifically for assessing MLLMs in video analysis. Setting itself apart from existing benchmarks, Video-MME offers four distinctive features: diversity in video types, duration in temporal dimension, breadth in data modalities, and quality in annotations. Encompassing six primary visual domains with 30 subfields to ensure broad scenario generalizability, Video-MME covers short-, medium-, and long-term videos ranging from 11 seconds to 1 hour to capture robust contextual dynamics. It also integrates multi-modal inputs beyond video frames such as subtitles and audios to showcase the all-round capabilities of MLLMs. Employing rigorous manual labeling by expert annotators enables precise and reliable model assessment. The dataset for Video-MME consists of 900 videos totaling 256 hours that were manually selected and annotated through repeated viewing to generate 2,700 question-answer pairs. This extensive dataset allows for thorough evaluation of various state-of-the-art MLLMs such as the GPT-4 series and Gemini 1.5 Pro alongside open-source image models like InternVL-Chat-V1.5 and video models like LLaVA-NeXT-Video. Our experiments reveal that Gemini 1.5 Pro stands out as the best-performing commercial model with significant superiority over open-source alternatives. However, these findings also highlight the ongoing need for further enhancements in handling longer sequences and multi-modal data within MLLMs. Through Video-MME's meticulous evaluation process and comprehensive dataset, we aim to drive advancements in leveraging Multi-modal Large Language Models for enhanced performance in video analysis tasks. For more information on this benchmark project, please visit our Project Page at https://video-mme.github.io.