# Multiview Transformers for Video Recognition

**Authors:** Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, Cordelia Schmid  
**arXiv:** [2201.04288](https://arxiv.org/abs/2201.04288)  
**Date:** 2022-01-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2201.04288v1/

## Key points

- Multiview Transformers for Video Recognition (MTV) is introduced as a model for reasoning at multiple spatiotemporal resolutions in video understanding.
- MTV utilizes separate encoders to represent different views of the input video and fuses information across views through lateral connections.
- Extensive ablation studies show that MTV consistently outperforms single-view counterparts in terms of accuracy and computational cost across various model sizes.
- MTV achieves state-of-the-art results on five standard datasets and improves performance with large-scale pretraining.
- Future research directions include exploring datasets beyond Kinetics and reducing dependence on supervised pretraining.
- The proposed method presents a promising approach to capturing multi-resolution temporal context in transformer architectures for video recognition tasks.

## Summary

The paper introduces Multiview Transformers for Video Recognition (MTV), a model that addresses the challenge of reasoning at multiple spatiotemporal resolutions in video understanding. While transformer architectures have advanced the state-of-the-art, they have not explicitly modeled different spatiotemporal resolutions. MTV overcomes this limitation by utilizing separate encoders to represent different views of the input video and fusing information across views through lateral connections. The authors conduct extensive ablation studies and demonstrate that MTV consistently outperforms single-view counterparts in terms of accuracy and computational cost across various model sizes. Additionally, MTV achieves state-of-the-art results on five standard datasets and further improves performance with large-scale pretraining. The paper concludes by acknowledging limitations and suggesting future research directions such as exploring datasets beyond Kinetics and reducing dependence on supervised pretraining. Overall, the proposed method presents a promising approach to capturing multi-resolution temporal context in transformer architectures for video recognition tasks.