# What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Models

**Authors:** Abdelrahman Abdelhamed, Mahmoud Afifi, Alec Go  
**arXiv:** [2405.15668](https://arxiv.org/abs/2405.15668)  
**Date:** 2024-05-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.15668v1/

## Key points

- Large language models (LLMs) are highly effective in image classification tasks
- Novel approach for zero-shot image classification using multimodal LLMs introduced by Abdelrahman Abdelhamed, Mahmoud Afifi, and Alec Go
- Comprehensive textual representations generated from input images using multimodal LLMs
- Fixed-dimensional features created within a cross-modal embedding space and fused for zero-shot classification through a linear classifier
- Single set of prompts used across all datasets, eliminating the need for prompt engineering for each dataset
- Method demonstrated an average accuracy gain of 4.1 percentage points across ten benchmarks compared to prior methods
- Substantial increase of 6.8 percentage points in accuracy on the ImageNet dataset
- Multimodal LLMs show significant potential in advancing computer vision tasks like zero-shot image classification

## Summary

, , , , 

In the realm of computer vision tasks, large language models (LLMs) have proven to be highly effective, particularly in image classification. A recent study by Abdelrahman Abdelhamed, Mahmoud Afifi, and Alec Go introduces a novel approach for zero-shot image classification using multimodal LLMs. By leveraging the capabilities of multimodal LLMs, the researchers generated comprehensive textual representations from input images. These representations were then used to create fixed-dimensional features within a cross-modal embedding space and fused together for zero-shot classification through a linear classifier. One notable aspect of this method is its ability to perform zero-shot classification without requiring prompt engineering for each dataset. Instead, a single set of prompts was used across all datasets, streamlining the process and enhancing efficiency. The efficacy of this approach was evaluated on multiple datasets, showcasing remarkable results that surpassed benchmark accuracy levels on various fronts. Specifically, when compared to prior methods, the proposed method demonstrated an average accuracy gain of 4.1 percentage points across ten benchmarks. Notably, on the challenging ImageNet dataset, there was a substantial increase of 6.8 percentage points in accuracy. These findings highlight the significant potential of multimodal LLMs in advancing computer vision tasks such as zero-shot image classification. Overall, this study sheds light on the promising prospects offered by integrating multimodal LLMs into image classification processes. By harnessing the power of language models in conjunction with visual data, researchers can achieve notable improvements in performance and accuracy levels, paving the way for enhanced capabilities in computer vision applications.