# Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Models

**Authors:** Rui Qian, Yeqing Li, Zheng Xu, Ming-Hsuan Yang, Serge Belongie, Yin Cui  
**arXiv:** [2207.07646](https://arxiv.org/abs/2207.07646)  
**Date:** 2022-07-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2207.07646v1/

## Key points

- Integration of vision and language models (VLMs) pre-trained on image-text pairs has shown promise
- Researchers including Rui Qian, Yeqing Li, Zheng Xu, Ming-Hsuan Yang, Serge Belongie, and Yin Cui expanded the paradigm by incorporating motion and audio elements in video data
- Introduction of MOV method for Multimodal Open-Vocabulary video classification
- Utilization of vision encoder from pre-trained VLMs to encode video frames, optical flow, and audio spectrograms
- Cross-modal fusion mechanism to effectively fuse multimodal inputs
- Significant performance improvements on benchmark datasets Kinetics-700 and VGGSound compared to existing methods and pre-trained VLM alone
- Enhanced accuracy on base classes and superior generalization capabilities on novel classes with MOV
- State-of-the-art results on zero-shot video classification benchmarks UCF and HMDB surpassing traditional zero-shot approaches and recent methods based on VLMs
- Commitment to sharing code and models with research community for further exploration in multimodal open-vocabulary video classification techniques

## Summary

In the realm of open-vocabulary visual recognition, the integration of vision and language models (VLMs) pre-trained on extensive image-text pairs has shown great promise. Building upon this foundation, a team of researchers including Rui Qian, Yeqing Li, Zheng Xu, Ming-Hsuan Yang, Serge Belongie, and Yin Cui have expanded this paradigm by incorporating motion and audio elements inherent in video data. Their innovative approach introduces \textbf{MOV}, a straightforward yet powerful method for Multimodal Open-Vocabulary video classification. The key innovation behind MOV lies in its utilization of the vision encoder from pre-trained VLMs with minimal adjustments to encode not only video frames but also optical flow and audio spectrograms. By seamlessly integrating these diverse modalities, MOV is able to capture a more comprehensive representation of the underlying visual content. To effectively fuse these multimodal inputs, the researchers have devised a cross-modal fusion mechanism that leverages complementary information across different modalities. Experimental evaluations conducted on benchmark datasets such as Kinetics-700 and VGGSound have demonstrated significant performance improvements when incorporating optical flow or audio modalities compared to existing methods and the pre-trained VLM alone. Particularly noteworthy is MOV's ability to enhance accuracy on base classes while also showcasing superior generalization capabilities on novel classes. Moreover, MOV has achieved state-of-the-art results on zero-shot video classification benchmarks such as UCF and HMDB, surpassing both traditional zero-shot approaches and recent methods based on VLMs. The researchers are committed to sharing their code and models with the research community, thereby facilitating further exploration and advancement in multimodal open-vocabulary video classification techniques.