# Learning Similarity between Scene Graphs and Images with Transformers

**Authors:** Yuren Cong, Wentong Liao, Bodo Rosenhahn, Michael Ying Yang  
**arXiv:** [2304.00590](https://arxiv.org/abs/2304.00590)  
**Date:** 2023-04-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.00590v1/

## Key points

- Authors address limitations of conventional scene graph generation evaluation metrics
- Triplet-oriented metrics like Recall@K fail to capture overall semantic information of scene graphs
- Proposed novel contrastive learning framework inspired by CLIP
- Framework includes graph Transformer and image Transformer aligning scene graphs and images in shared latent space
- Introduction of graph serialization technique to enhance understanding of scene graph structures
- Introduction of R-Precision as a new evaluation metric for image retrieval accuracy in scene graph generation
- Establishment of new benchmarks for datasets like Visual Genome and Open Images using proposed framework
- Effectiveness demonstrated through experiments, highlighting the promise of the graph Transformer as an encoder for scene graphs

## Summary

In their paper titled "Learning Similarity between Scene Graphs and Images with Transformers," authors Yuren Cong, Wentong Liao, Bodo Rosenhahn, and Michael Ying Yang address the limitations of conventional scene graph generation evaluation metrics. They highlight that triplet-oriented metrics like (mean) Recall@K fail to capture the overall semantic information of scene graphs and do not effectively measure the similarity between images and generated scene graphs. This hinders the usability of scene graphs in downstream tasks. To overcome this challenge, the authors propose a novel contrastive learning framework inspired by Contrastive Language-Image Pre-training (CLIP). Their framework includes a graph Transformer and an image Transformer designed to align scene graphs and corresponding images in a shared latent space. To enhance the graph Transformer's ability to understand scene graph structures and extract meaningful features, they introduce a graph serialization technique that transforms scene graphs into structured sequences. The authors introduce R-Precision as a new evaluation metric for measuring image retrieval accuracy in scene graph generation. They establish new benchmarks for datasets like Visual Genome and Open Images using their proposed framework. Through a series of experiments, they demonstrate the effectiveness of the graph Transformer as a promising encoder for scene graphs. Overall, this research contributes significantly to improving the evaluation and utility of scene graphs by introducing a comprehensive framework that enhances the alignment between images and corresponding scene graphs through advanced transformer models. The findings pave the way for more accurate and semantically rich representations of visual scenes in various applications.