# ViViT: A Video Vision Transformer

**Authors:** Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, Cordelia Schmid  
**arXiv:** [2103.15691](https://arxiv.org/abs/2103.15691)  
**Date:** 2021-03-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2103.15691v2/

## Key points

- Paper presents pure-transformer based models for video classification
- Model extracts spatio-temporal tokens from input videos and encodes them using transformer layers
- Efficient variants of the model introduced to handle long sequences of tokens in videos
- Effective regularization techniques used during training and pretrained image models leveraged to train on small datasets
- Comprehensive ablation studies conducted
- Achieved state-of-the-art results on various video classification benchmarks including Kinetics 400 and 600, Epic Kitchens, Something-Something v2, and Moments in Time
- Outperforms prior methods based on deep 3D convolutional networks
- Code for the model released at ICCV 2021 available at https://github.com/google-research/scenic/tree/main/scenic/projects/vivit

## Summary

The paper titled "ViViT: A Video Vision Transformer" presents pure-transformer based models for video classification, building upon the success of such models in image classification. The authors propose a model that extracts spatio-temporal tokens from input videos and encodes them using transformer layers. To handle the long sequences of tokens in videos, they introduce efficient variants of their model that factorize the spatial and temporal dimensions of the input. Despite transformer-based models typically requiring large training datasets, the authors demonstrate effective regularization techniques during training and leverage pretrained image models to train on comparatively small datasets. The authors conduct comprehensive ablation studies and achieve state-of-the-art results on various video classification benchmarks, including Kinetics 400 and 600, Epic Kitchens, Something-Something v2, and Moments in Time. Their approach outperforms prior methods based on deep 3D convolutional networks. To facilitate further research in this area, the authors release code for their model which was published at ICCV 2021 and is available at https://github.com/google-research/scenic/tree/main/scenic/projects/vivit. Overall, this paper introduces an innovative approach to video classification using transformer-based models which are capable of achieving state-of-the art results on multiple benchmarks with comparatively small datasets due to effective regularization techniques and leveraging pretrained image models during training. Furthermore, to facilitate further research in this area, the authors have released code for their model which is available online.