# CLIPScore: A Reference-free Evaluation Metric for Image Captioning

**Authors:** Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, Yejin Choi  
**arXiv:** [2104.08718](https://arxiv.org/abs/2104.08718)  
**Date:** 2021-04-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2104.08718v1/

## Key points

- Evaluation of machine-generated captions traditionally relies on reference-based metrics
- Humans assess caption quality in a reference-free manner
- CLIP (Contrastive Language-Image Pretraining) can be used for automatic evaluation without references
- CLIPScore is a new reference-free metric that achieves high correlation with human judgments
- CLIPScore outperforms existing reference-based metrics like CIDEr and SPICE
- CLIPScore complements existing metrics by emphasizing image-text compatibility
- RefCLIPScore, a reference augmented version, achieves even higher correlation with human judgments
- Case studies show CLIPScore performs well in clip art images and alt text rating tasks
- CLIPScore is relatively weaker in news captions requiring richer contextual knowledge
- CLIPScore introduces an innovative approach to evaluating image captioning using the CLIP model

## Summary

In the field of image captioning, the evaluation of machine-generated captions has traditionally relied on reference-based metrics, where the machine captions are compared to human-written captions. However, this approach differs from how humans themselves assess the quality of captions, as they do so in a reference-free manner. In this paper titled "CLIPScore: A Reference-free Evaluation Metric for Image Captioning," authors Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras and Yejin Choi present a surprising empirical finding. The authors demonstrate that CLIP (Contrastive Language-Image Pretraining), a cross-modal model pretrained on 400 million image-caption pairs from the web can be effectively used for automatic evaluation of image captioning without relying on references. They introduce a new reference-free metric called CLIPScore and show through experiments across multiple datasets that it achieves the highest correlation with human judgments. In fact, CLIPScore outperforms existing reference-based metrics such as CIDEr and SPICE. The authors also conduct information gain experiments to highlight the complementary nature of CLIPScore with respect to existing reference-based metrics which primarily focus on text-text similarities. They find that CLIPScore's emphasis on image-text compatibility makes it particularly suitable for evaluating literal description tasks. Additionally, they present a reference augmented version called RefCLIPScore which achieves even higher correlation with human judgments. Furthermore, the paper includes several case studies which showcase domains where CLIPScore performs well such as clip art images and alt text rating tasks. However it also identifies areas where CLIPScore is relatively weaker compared to reference based metrics - for example news captions that require richer contextual knowledge. Overall this research introduces an innovative approach to evaluating image captioning using CLIPScore - a novel reference free metric based on the CLIP model pretrained on large scale data. The findings highlight its effectiveness in capturing image text compatibility and its potential to enhance the evaluation process in various applications.