# Rethinking Benchmarks for Cross-modal Image-text Retrieval

**Authors:** Weijing Chen, Linli Yao, Qin Jin  
**arXiv:** [2304.10824](https://arxiv.org/abs/2304.10824)  
**Date:** 2023-04-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.10824v1/

## Key points

- Image-text retrieval is an important aspect of information retrieval.
- The primary challenge in this task is cross-modal semantic understanding and matching.
- Recent research has focused on fine-grained cross-modal semantic matching, made possible by large-scale multimodal pretraining models.
- State-of-the-art models like X-VLM have achieved near-perfect performance on widely used image-text retrieval benchmarks like MSCOCO Test 5K and Flickr30K Test 1K.
- However, these benchmarks are insufficient to assess the true capability of models on fine-grained cross modal semantic matching due to the coarse grained nature of a large portion of images and texts in these benchmarks.
- To address this issue, the authors propose renovating the coarse grained images and texts in existing benchmarks to establish improved benchmarks called MSCOCO FG and Flickr30K FG.
- The authors evaluate representative image text retrieval models on their new benchmarks to demonstrate the effectiveness of their method while also analyzing model capabilities for fine-grained semantic comprehension through extensive experiments.
- The results reveal that even state-of-the-art models have much room for improvement in fine-grained semantic understanding, particularly when distinguishing attributes of close objects in images.
- This paper's contribution lies in its proposal for improved benchmarks that better reflect real-world scenarios where fine-grained cross modal semantic matching is necessary, highlighting areas where current state-of-the-art models fall short and providing a foundation for future research in this field.

## Summary

The field of image-text retrieval has garnered significant attention as a crucial aspect of information retrieval. The primary challenge in this task is cross-modal semantic understanding and matching. Recent research has focused on fine-grained cross-modal semantic matching, which has been made possible by the prevalence of large-scale multimodal pretraining models. State-of-the-art models such as X-VLM have achieved near-perfect performance on widely used image-text retrieval benchmarks like MSCOCO Test 5K and Flickr30K Test 1K. However, upon reviewing these benchmarks, it becomes clear that they are insufficient to assess the true capability of models on fine-grained cross modal semantic matching due to the coarse grained nature of a large portion of images and texts in these benchmarks. To address this issue, the authors propose renovating the coarse grained images and texts in existing benchmarks to establish improved benchmarks called MSCOCO FG and Flickr30K FG. On the image side, they enlarge the original image pool by adopting more similar images. On the text side, they propose a novel semi automatic renovation approach to refine coarse grained sentences into finer grained ones with little human effort. The authors evaluate representative image text retrieval models on their new benchmarks to demonstrate the effectiveness of their method while also analyzing model capabilities for fine grained semantic comprehension through extensive experiments. The results reveal that even state of the art models have much room for improvement in fine grained semantic understanding, particularly when distinguishing attributes of close objects in images. The code and improved benchmark datasets are publicly available at https://github.com/cwj1412/MSCOCO Flikcr30K_FG , with hopes that it will inspire further research on cross modal retrieval. This paper's contribution lies in its proposal for improved benchmarks that better reflect real world scenarios where fine grained cross modal semantic matching is necessary, highlighting areas where current state of the art models fall short and providing a foundation for future research in this field.