# Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation

**Authors:** Sara Sarto, Manuele Barraco, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara  
**arXiv:** [2303.12112](https://arxiv.org/abs/2303.12112)  
**Date:** 2023-03-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.12112v3/

## Key points

- Authors Sara Sarto, Manuele Barraco, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara introduce a novel evaluation metric called Positive-Augmented Contrastive learning Score (PAC-S) for image captioning.
- PAC-S combines contrastive visual-semantic space learning with the incorporation of generated images and text on curated data to enhance existing reference-based and reference-free metrics.
- Through experiments across multiple datasets, PAC-S achieves the highest correlation with human judgments for both images and videos, surpassing existing metrics like CIDEr, SPICE, and CLIP-Score.
- The study evaluates the system-level correlation of PAC-S when applied to popular image captioning approaches while examining the impact of utilizing different cross-modal features.
- Highlighted as a highlight paper at CVPR 2023, this research significantly advances evaluation methods in image and video captioning.
- The availability of their source code and trained models on GitHub ensures transparency and reproducibility in further studies within this domain.

## Summary

In their paper titled "Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation," authors Sara Sarto, Manuele Barraco, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara introduce a novel evaluation metric called Positive-Augmented Contrastive learning Score (PAC-S) for image captioning. The proposed method combines contrastive visual-semantic space learning with the incorporation of generated images and text on curated data to improve upon existing reference-based and reference-free metrics. Through experiments across multiple datasets, the researchers demonstrate that PAC-S achieves the highest correlation with human judgments for both images and videos. This new metric surpasses existing metrics like CIDEr and SPICE as well as CLIP-Score. Additionally, the study evaluates the system-level correlation of PAC-S when applied to popular image captioning approaches while assessing the impact of utilizing different cross-modal features. Highlighted as a highlight paper at CVPR 2023, this research contributes significantly to advancing evaluation methods in image and video captioning. The availability of their source code and trained models on GitHub ensures transparency and reproducibility in further studies within this domain.