# mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video

**Authors:** Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, Guohai Xu, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou  
**arXiv:** [2302.00402](https://arxiv.org/abs/2302.00402)  
**Date:** 2023-02-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.00402v1/

## Key points

- Significant convergence of language, vision, and multi-modal pretraining in AI field
- Development of large-scale pre-trained foundation models spanning various domains
- Transformer architecture (e.g., T5, OFA, Flamingo) crucial in unifying tasks and modalities
- Challenges with modality entanglement when dealing with multiple modalities in a single network
- Introduction of mPLUG-2 paradigm for multi-modal pretraining to mitigate effects of modality entanglement
- mPLUG-2 features a multi-module composition network for effective collaboration and disentanglement of different modalities
- Achieves state-of-the-art or competitive results on over 30 downstream tasks encompassing multi-modal and uni-modal tasks
- Demonstrates new state-of-the-art results on video QA and video caption tasks with smaller model size and data scale compared to existing models
- Strong zero-shot transferability on vision-language and video-language tasks
- Modularized multi-modal foundation model across text, image, and video domains with enhanced collaboration between modalities

## Summary

In recent years, there has been a significant convergence of language, vision, and multi-modal pretraining in the field of artificial intelligence. This trend has led to the development of large-scale pre-trained foundation models that span various domains such as language, vision, and multi-modality. The emergence of Transformer architecture has played a crucial role in this convergence, with models like T5, OFA, and Flamingo unifying tasks and modalities within a sequence-to-sequence generation framework. However, challenges arise when dealing with multiple modalities within a single network due to modality entanglement. Different modalities may interfere with each other, especially when there are numerous modalities and tasks involved. To address this issue, a new paradigm called mPLUG-2 has been introduced. <br><br>
 for multi-modal pretraining  aims to benefit from  while mitigating the effects of . Unlike traditional paradigms that rely solely on sequence-to-sequence generation or encoder-based instance discrimination,mPLUG-2 introduces a multi-module composition network. This network shares common universal modules for modality collaboration and disentangles different modality modules to effectively deal with modality entanglement. The flexibility of mPLUG-2 allows for the selection of different modules for various understanding and generation tasks across all modalities including text,image,and video.Empirical studies have shown that mPLUG-2 achieves state-of-the-art or competitive results on over 30 downstream tasks encompassing multi-modal tasks such as image-text and video-text understanding and generation as well as uni-modal tasks focusing on text-only, image-only, and video-only understanding. Notably,mPLUG-2 has demonstrated new state-of-the-art results on challenging video QA and video caption tasks with significantly smaller model size and data scale compared to existing models. Additionally,mPLUG-2 exhibits strong zero-shot transferability on vision-language and video-language tasks. Overall,mPLUG-2 presents a modularized multi-modal foundation model across text,image,and video domains. Its innovative design allows for enhanced collaboration between different modalities while effectively addressing the complexities associated with modality entanglement. The release of code and models on GitHub further enhances accessibility to this cutting-edge technology for researchers in the field.