# VideoMamba: State Space Model for Efficient Video Understanding

**Authors:** Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, Yu Qiao  
**arXiv:** [2403.06977](https://arxiv.org/abs/2403.06977)  
**Date:** 2024-03-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.06977v1/

## Key points

- VideoMamba is a novel model that adapts the Mamba architecture to the video domain.
- It addresses local redundancy and global dependencies in video understanding, overcoming limitations of existing 3D convolution neural networks and video transformers.
- VideoMamba utilizes a linear-complexity operator for efficient long-term modeling essential for high-resolution video comprehension.
- The proposed VideoMamba exhibits four core abilities: scalability in the visual domain without extensive dataset pretraining, sensitivity for recognizing short-term actions, superiority in long-term video understanding compared to traditional models, and compatibility with other modalities showcasing robustness in multi-modal contexts.
- VideoMamba sets a new benchmark for comprehensive video understanding with superior zero-shot video-text retrieval performance compared to existing models like UMT based on ViT.
- It excels in handling multi-modal tasks efficiently and demonstrates significant improvement in datasets featuring longer videos and complex scenarios.
- VideoMamba operates six times faster than TimeSformer and requires significantly less GPU memory for 64-frame videos while interpreting long videos effectively.
- Its adaptability with other modalities is evident through improved performance in video-text retrievals compared to ViT, especially in complex scenarios.

## Summary

This work introduces VideoMamba, a novel model that adapts the Mamba architecture to the video domain. Addressing the dual challenges of local redundancy and global dependencies in video understanding, VideoMamba overcomes limitations of existing 3D convolution neural networks and video transformers by utilizing a linear-complexity operator for efficient long-term modeling essential for high-resolution video comprehension. The proposed VideoMamba exhibits four core abilities: scalability in the visual domain without extensive dataset pretraining through a self-distillation technique, sensitivity for recognizing short-term actions with fine-grained motion differences, superiority in long-term video understanding compared to traditional feature-based models, and compatibility with other modalities showcasing robustness in multi-modal contexts. Through these advantages, VideoMamba sets a new benchmark for comprehensive video understanding. In extensive evaluations across prominent benchmarks such as MSRVTT, DiDeMo, ActivityNet, LSMDC, and MSVD, VideoMamba demonstrates superior zero-shot video-text retrieval performance compared to existing models like UMT based on ViT. Particularly notable is its efficiency and scalability in handling multi-modal tasks and its significant improvement in datasets featuring longer videos and complex scenarios. Furthermore, VideoMamba excels in interpreting long videos by operating six times faster than TimeSformer and requiring significantly less GPU memory for 64-frame videos. Its adaptability with other modalities is evident through improved performance in video-text retrievals compared to ViT, especially in complex scenarios. Overall  shows immense potential in understanding both short-term and long-term video content across various datasets like K400,SthSthV2,Breakfast ,COIN,and LVU.With its efficiency and effectiveness demonstrated through thorough experiments, is positioned to become a cornerstone in the field of long-video comprehension. All code and models are openly available at https://github.com/OpenGVLab/VideoMamba to support future research endeavors.