# No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance

**Authors:** Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip H. S. Torr, Adel Bibi, Samuel Albanie, Matthias Bethge  
**arXiv:** [2404.04125](https://arxiv.org/abs/2404.04125)  
**Date:** 2024-04-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.04125v1/

## Key points

- Investigated performance of multimodal models on downstream concepts in relation to frequency in pretraining datasets
- Multimodal models do not exhibit "zero-shot" generalization and require exponentially more data for linear improvements
- Sample inefficient log-linear scaling trend observed even with sample-level similarity control and testing on synthetic data
- Models underperformed significantly on long-tailed concepts compared to ImageNet, with higher capacity models showing some improvement
- Need for better strategies for sample-efficient learning in multimodal models, especially for rare concepts

## Summary

In this study, we investigated the performance of multimodal models on downstream concepts in relation to the frequency of these concepts in their pretraining datasets. We conducted experiments using 34 models and five standard pretraining datasets, generating a vast amount of data artifacts for analysis. Our findings revealed that multimodal models do not exhibit "zero-shot" generalization as previously thought. Instead, they require exponentially more data to achieve linear improvements in downstream performance. This sample inefficient log-linear scaling trend persisted even when controlling for sample-level similarity between pretraining and downstream datasets and testing on purely synthetic data distributions. Furthermore, our classification experiments showed that all models underperformed significantly on long-tailed concepts compared to ImageNet, with higher capacity models showing some improvement. This benchmark dataset highlights the need for better strategies for sample-efficient learning in multimodal models. Our study adds to existing research on large-scale datasets and their impact on model performance across various tasks. Prior work has emphasized the importance of data for improving model generalization and performance while also addressing issues such as concept redundancy and biases in pretraining datasets. By focusing on long-tailed concepts in pretraining data distributions, our work sheds light on the challenges faced by current multimodal models in comprehending and representing rare concepts effectively. This indicates a need for further exploration and development in this field.