# Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratings

**Authors:** Olivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kajić, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Chris Knutsen, Cyrus Rashtchian, Jordi Pont-Tuset, Aida Nematzadeh  
**arXiv:** [2404.16820](https://arxiv.org/abs/2404.16820)  
**Date:** 2024-04-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.16820v1/

## Key points

- Growing need for text-to-image (T2I) generative models to align generated images closely with given prompts
- Recent study introduces Gecko2K benchmark to categorize prompts into sub-skills and identify challenging areas for T2I models
- Analysis of over 100K human ratings across templates and models reveals differences due to prompt ambiguity, metric quality, and model discrepancies
- Introduction of a new QA-based auto-eval metric that shows better correlation with human ratings compared to existing metrics
- Development of Gecko(S) prompt set with fine-grained skills coverage for identifying T2I model failures
- Importance of using reliable prompts with high inter-annotator agreement for consistent model ordering
- Fine-grained annotation templates yield more consistent results compared to coarse-grained ones
- Significance of standardizing model evaluation processes by considering benchmark selection and annotation template quality

## Summary

, , , , 

In the realm of text-to-image (T2I) generative models, there is a growing need to ensure that generated images align closely with given prompts. Previous efforts have focused on evaluating T2I alignment through metrics, benchmarks, and human judgement templates. However, the quality and reliability of these components have not been systematically measured. This gap is addressed in a recent study that delves into auto-eval metrics and human templates to provide a more comprehensive understanding. The study introduces Gecko2K, a detailed benchmark that categorizes prompts into sub-skills to pinpoint challenging areas for T2I models. By gathering over 100K human ratings across four templates and four T2I models, the research sheds light on where differences arise due to prompt ambiguity versus metric and model quality discrepancies. Additionally, a new QA-based auto-eval metric is introduced, showcasing better correlation with human ratings compared to existing metrics. Key contributions include the development of Gecko(S), a discriminative prompt set with fine-grained skills coverage for identifying T2I model failures. The analysis highlights the impact of annotation templates on model evaluation and emphasizes the importance of using reliable prompts with high inter-annotator agreement for consistent model ordering. Furthermore, findings suggest that fine-grained annotation templates yield more consistent results compared to coarse-grained ones. Overall, the study underscores the significance of standardizing model evaluation processes by considering both benchmark selection and annotation template quality. While the proposed metric shows promise for reliable model comparisons, future directions may involve incorporating confidence thresholds alongside metric scores. Anecdotal evidence suggests that annotators spend more time rating prompt-image pairs using certain templates, indicating potential variations in evaluation efficiency based on template type.