# MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition

**Authors:** Jiawei Chen, Chiu Man Ho  
**arXiv:** [2108.09322](https://arxiv.org/abs/2108.09322)  
**Date:** 2021-08-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2108.09322v1/

## Key points

- Authors Jiawei Chen and Chiu Man Ho introduce MM-ViT, a transformer-based approach for compressed video action recognition
- MM-ViT operates in the compressed video domain and uses modalities like I-frames, motion vectors, residuals, and audio waveforms
- Scalable model variants are proposed to handle spatiotemporal tokens from multiple modalities by factorizing self-attention across space, time, and modality dimensions
- Three distinct cross-modal attention mechanisms are developed to explore rich inter-modal interactions within the transformer building block
- MM-ViT outperforms state-of-the-art video transformers in efficiency and accuracy on UCF-101, Something-Something-v2, Kinetics-600 benchmarks
- It performs comparably or better than CNN counterparts using computationally-heavy optical flow
- MM-ViT efficiently utilizes multiple modalities within a transformer framework to enhance video action recognition tasks
- This research demonstrates the potential for further advancements in multi-modal deep learning models for complex visual tasks

## Summary

In their paper titled "MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition," authors Jiawei Chen and Chiu Man Ho introduce a novel transformer-based approach for video action recognition. The Multi-Modal Video Transformer (MM-ViT) operates exclusively in the compressed video domain and leverages various modalities such as I-frames, motion vectors, residuals, and audio waveforms. To handle the large number of spatiotemporal tokens extracted from multiple modalities, the authors propose scalable model variants that factorize self-attention across space, time, and modality dimensions. Additionally, they explore rich inter-modal interactions by developing three distinct cross-modal attention mechanisms that can seamlessly integrate into the transformer building block. Through extensive experiments on three public action recognition benchmarks (UCF-101, Something-Something-v2, Kinetics-600), MM-ViT outperforms state-of-the-art video transformers in terms of efficiency and accuracy. It also performs comparably or even better than CNN counterparts that utilize computationally-heavy optical flow. This research showcases the effectiveness of MM-ViT in enhancing video action recognition tasks by efficiently utilizing multiple modalities within a transformer framework. Overall, this approach not only improves performance but also highlights the potential for further advancements in multi-modal deep learning models for complex visual tasks.