# Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

**Authors:** Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, Xinggang Wang  
**arXiv:** [2401.09417](https://arxiv.org/abs/2401.09417)  
**Date:** 2024-01-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.09417v1/

## Key points

- State space models (SSMs) with efficient hardware-aware designs, like Mamba, have shown promise in long sequence modeling
- Vision Mamba (Vim) is a new approach that builds efficient and generic vision backbones using SSMs
- Vim incorporates bidirectional Mamba blocks into the architecture instead of relying solely on self-attention for visual representation learning
- Vim marks image sequences with position embeddings and compresses visual representations using bidirectional state space models
- Vim achieves higher performance compared to well-established vision transformers like DeiT on tasks such as ImageNet classification, COCO object detection, and ADE20k semantic segmentation
- Vim demonstrates significantly improved computation and memory efficiency, being 2.8 times faster than DeiT and saving 86.8% GPU memory during batch inference on high-resolution images (1248x1248 resolution)
- Vim overcomes computation and memory constraints while performing Transformer-style understanding for high-resolution images
- Code for implementing Vim can be found at https://github.com/hustvl/Vim
- Vim offers a promising solution for efficient visual representation learning in computer vision tasks and has the potential to become the next-generation backbone for vision foundation models

## Summary

In recent years, state space models (SSMs) with efficient hardware-aware designs, such as Mamba, have demonstrated promising results in long sequence modeling. Expanding on this idea, researchers propose a new approach called Vision Mamba (Vim) to build efficient and generic vision backbones using SSMs. The authors argue that relying solely on self-attention for visual representation learning is not necessary and introduce Vim which incorporates bidirectional Mamba blocks into the architecture. Vim marks image sequences with position embeddings and compresses visual representations using bidirectional state space models. The proposed backbone achieves higher performance compared to well-established vision transformers like DeiT on tasks such as ImageNet classification, COCO object detection, and ADE20k semantic segmentation. Additionally, Vim demonstrates significantly improved computation and memory efficiency. For instance, Vim is 2.8 times faster than DeiT and saves 86.8% GPU memory when performing batch inference on high-resolution images (1248x1248 resolution). These results highlight Vim's ability to overcome computation and memory constraints while performing Transformer-style understanding for high-resolution images. The authors provide code for Vim implementation at https://github.com/hustvl/Vim. With its potential to become the next-generation backbone for vision foundation models, Vim offers a promising solution for efficient visual representation learning in computer vision tasks.