# A Practitioner's Guide to Continual Multimodal Pretraining

**Authors:** Karsten Roth, Vishaal Udandarao, Sebastian Dziadzio, Ameya Prabhu, Mehdi Cherti, Oriol Vinyals, Olivier Hénaff, Samuel Albanie, Matthias Bethge, Zeynep Akata  
**arXiv:** [2408.14471](https://arxiv.org/abs/2408.14471)  
**Date:** 2024-08-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2408.14471v1/

## Key points

- Multimodal foundation models face the challenge of model obsolescence despite extensive pretraining on vast datasets.
- Research into continual pretraining has focused on infrequent updates on large-scale new data or frequent sample-level updates.
- Practical deployment often requires adaptation to specific subdomains, tasks, or concepts throughout a model's lifecycle, creating a complex landscape for continual model updates.
- FoMo-in-Flux is introduced as a benchmark for continual multimodal pretraining with realistic compute constraints and practical deployment requirements.
- FoMo-in-Flux is constructed over 63 datasets with diverse visual and semantic coverage, serving as a test bed for exploring practical continual pretraining nuances.
- The investigation includes data mixtures, stream orderings reflecting real-world situations, method-centric approaches like fine-tuning and parameter-efficient updates, meta learning rate schedules, mechanistic design choices, and the impact of model and compute scaling.
- A practitioner's guide to continual multimodal pretraining for real-world deployment is provided based on insights gained from the exploration.
- The benchmark and accompanying code are available for further research and application in the field.

## Summary

In the realm of multimodal foundation models that bridge the gap between vision and language, the challenge of model obsolescence over time looms large despite their extensive pretraining on vast datasets. To address this issue, research into continual pretraining has predominantly focused on two scenarios: infrequent updates on large-scale new data or frequent sample-level updates. However, practical deployment often necessitates adaptation to specific subdomains, tasks, or concepts throughout a model's lifecycle, presenting a complex landscape for continual model updates. In response to these challenges, this work introduces FoMo-in-Flux, a benchmark for continual multimodal pretraining designed with realistic compute constraints and practical deployment requirements. Constructed over 63 datasets with diverse visual and semantic coverage, FoMo-in-Flux serves as a test bed for exploring the nuances of practical continual pretraining. The investigation delves into various perspectives including data mixtures and stream orderings mirroring real-world deployment situations, method-centric approaches ranging from fine-tuning to parameter-efficient updates and model merging, meta learning rate schedules, mechanistic design choices, as well as the impact of model and compute scaling. Through comprehensive insights gained from this exploration, a practitioner's guide to continual multimodal pretraining for real-world deployment is provided. This guide offers valuable guidance on navigating the complexities of updating models in dynamic environments where adaptation to specific subdomains or tasks is crucial. The benchmark along with accompanying code is made available for further research and application in the field.