# Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts

**Authors:** Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, Neil Houlsby  
**arXiv:** [2206.02770](https://arxiv.org/abs/2206.02770)  
**Date:** 2022-06-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2206.02770v1/

## Key points

- The Language-Image MoE (LIMoE) is a sparse mixture of experts model capable of multimodal learning.
- LIMoE accepts both images and text simultaneously and is trained using a contrastive loss.
- The model uses expert layers to learn an appropriate partitioning of modalities, but new challenges arise, such as training stability and balanced expert utilization, for which the authors propose an entropy-based regularization scheme.
- LIMoE-L/16 trained comparably to CLIP-L/14 achieves 78.6% zero-shot ImageNet accuracy (vs. 76.2%), and when further scaled to H/14 with additional data, it achieves 84.1%, comparable to state-of-the-art methods that use larger custom per-modality backbones and pre-training schemes.
- In related work, unimodal task-specific neural networks have been researched extensively, with increasing convergence towards Transformer-based architectures for both NLP and Computer Vision.
- Multimodal models aim to process multiple types of data using a single neural network.
- The paper builds on deep Sparse Mixture of Experts models studied independently in Computer Vision and NLP contexts for transfer learning purposes.
- Contrastive learning has been widely researched in self-supervised regimes but also in supervised regimes for aligned data from multiple modalities.
- The authors propose that LIMoE is naturally a good candidate for efficient, large scale multimodal foundation models.

## Summary

This paper introduces the Language-Image MoE (LIMoE), a sparse mixture of experts model capable of multimodal learning. While large sparsely-activated models have achieved excellent performance in various domains, they are typically trained on a single modality at a time. LIMoE accepts both images and text simultaneously and is trained using a contrastive loss. The model uses expert layers to learn an appropriate partitioning of modalities, but new challenges arise, such as training stability and balanced expert utilization, for which the authors propose an entropy-based regularization scheme. The paper presents remarkable performance improvements over dense models of equivalent computational cost across multiple scales. LIMoE-L/16 trained comparably to CLIP-L/14 achieves 78.6% zero-shot ImageNet accuracy (vs. 76.2%), and when further scaled to H/14 with additional data, it achieves 84.1%, comparable to state-of-the-art methods that use larger custom per-modality backbones and pre-training schemes. The authors also analyze the quantitative and qualitative behavior of LIMoE and demonstrate phenomena such as differing treatment of the modalities and the organic emergence of modality-specific experts. In related work, unimodal task-specific neural networks have been researched extensively, with increasing convergence towards Transformer-based architectures for both NLP and Computer Vision. Multimodal models aim to process multiple types of data using a single neural network. Many approaches fuse modalities or co-train on distinct tasks without aligning or fusing representations. The paper builds on deep Sparse Mixture of Experts models studied independently in Computer Vision and NLP contexts for transfer learning purposes. These models use a learned gating mechanism whereby only a subset of K experts out of E are activated for a given input Contrastive learning has been widely researched in self-supervised regimes but also in supervised regimes for aligned data from multiple modalities. However, the authors are not aware of previous research using a single model to process both images and texts for contrastive learning, neither with dense nor with sparse models. The paper concludes by discussing potential harms of large scale models, contrastive models, and web scale multimodal data. The authors propose that LIMoE is naturally a good candidate for efficient, large scale multimodal foundation models.