# GeneCIS: A Benchmark for General Conditional Image Similarity

**Authors:** Sagar Vaze, Nicolas Carion, Ishan Misra  
**arXiv:** [2306.07969](https://arxiv.org/abs/2306.07969)  
**Date:** 2023-06-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.07969v1/

## Key points

- Models should be able to adapt to different notions of similarity dynamically
- The authors propose the GeneCIS benchmark for General Conditional Image Similarity to measure models' ability to adapt to a range of similarity conditions
- The benchmark is designed for zero-shot evaluation only and considers an open-set of similarity conditions
- Baselines from powerful CLIP models struggle on GeneCIS and performance on the benchmark is weakly correlated with ImageNet accuracy
- The authors propose a simple, scalable solution based on automatically mining information from existing image-caption datasets to address this issue
- Their method offers a substantial boost over the baselines on GeneCIS and further improves zero-shot performance on related image retrieval benchmarks
- Their model surpasses state-of-the-art supervised models on MIT-States even though evaluated zero-shot
- Statistics of evaluations are shown in Table 1 including number of retrieval templates and gallery images as well as carefully constructed benchmarks with only one 'positive' image among targets with gallery sizes between 10 and 15 images.
- Distribution of objects and attributes specified in the conditions are shown in Figure 3 noting that their space of conditions spans a long tail of over 400 attributes and 100 objects.
- Strongest ViT-B/16 model's results are reported in Table 2 and scaling up mined triplets used for training improves performance shown in Figure 5.
- Different CLIP backbones' impact on their model's performance is explored in Figure 6.
- This paper proposes an important but understudied problem in computer vision: General Conditional Image Similarity.
- -The proposed benchmark evaluates an open set of similarity conditions and is designed for zero shot testing only.
- -The authors propose a way forward for scalably training conditional similarity models which mines information from widely available image caption datasets.
- -Their method not only boosts performance over all baselines on GeneCIS but also provides substantial zero shot gains on related image retrieval tasks.

## Summary

In this paper, the authors argue that models should be able to adapt to different notions of similarity dynamically. To address this issue, they propose the GeneCIS benchmark for General Conditional Image Similarity which measures models' ability to adapt to a range of similarity conditions. The benchmark is designed for zero-shot evaluation only and considers an open-set of similarity conditions. The authors find that baselines from powerful CLIP models struggle on GeneCIS and that performance on the benchmark is weakly correlated with ImageNet accuracy. To address this issue, the authors propose a simple, scalable solution based on automatically mining information from existing image-caption datasets. They find their method offers a substantial boost over the baselines on GeneCIS and further improves zero-shot performance on related image retrieval benchmarks. In fact, though evaluated zero-shot, their model surpasses state-of-the-art supervised models on MIT-States. The authors also show statistics of the evaluations in Table 1 including the number of retrieval templates and number of gallery images as well as carefully constructing the benchmarks such that there is only one ‘positive’ image among the targets with gallery sizes between 10 and 15 images. They further show the distribution of objects and attributes specified in the conditions in Figure 3 noting that their space of conditions spans a long tail of over 400 attributes and 100 objects. The authors report their strongest ViT-B/16 model's results in Table 2 and show how scaling up mined triplets used for training improves performance in Figure 5. They also explore different CLIP backbones' impact on their model's performance in Figure 6. In conclusion, this paper proposes an important but understudied problem in computer vision: General Conditional Image Similarity. The proposed benchmark evaluates an open set of similarity conditions and is designed for zero shot testing only. The authors propose a way forward for scalably training conditional similarity models which mines information from widely available image caption datasets. Their method not only boosts performance over all baselines on GeneCIS but also provides substantial zero shot gains on related image retrieval tasks.