# TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification

**Authors:** Qinying Liu, Kecheng Zheng, Wu Wei, Zhan Tong, Yu Liu, Wei Chen, Zilei Wang, Yujun Shen  
**arXiv:** [2312.14149](https://arxiv.org/abs/2312.14149)  
**Date:** 2023-12-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.14149v1/

## Key points

- Proposal of a simple yet effective approach to improve alignment between image and text features in vision-language models
- Addressing the problem of coarse alignment in vision encoders struggling to localize attribute-specified objects
- Use of parsing objects and attributes from image-text pairs as supervision signals
- Introduction of a multi-tag classification loss in addition to the commonly used image-text contrastive loss
- Experimental results demonstrating an average improvement of 3.65% over existing alternatives on semantic segmentation datasets
- Ablation study comparing different loss functions for mitigating tag imbalance, with balanced softmax loss outperforming other alternatives
- Evaluation of the proposed method's versatility across different visual encoders, including GroupViT-based encoder
- Significant performance improvements observed on both ViT-based CLIP and GroupViT encoders, highlighting generalization ability
- Visualization results showing accurate localization of attribute-specified objects with attribute supervision
- Overall, presentation of a simple yet effective approach for better aligning image and text features in vision-language models, leading to improved performance on semantic segmentation tasks.

## Summary

In this paper, the authors propose a simple yet effective approach to improve the alignment between image and text features in vision-language models. The existing problem of coarse alignment, where the vision encoder struggles to localize attribute-specified objects, is addressed. The proposed method involves parsing objects and attributes from image-text pairs, which are highly likely to exist in the image. This parsing pipeline is fully automatic and scalable. By using these parsed semantics as supervision signals, the authors complement the commonly used image-text contrastive loss with a multi-tag classification loss. Experimental results on various semantic segmentation datasets demonstrate an average improvement of 3.65% over existing alternatives. The authors also conduct an ablation study on long-tailed training loss, comparing different loss functions aimed at mitigating tag imbalance. The balanced softmax loss is found to outperform other alternatives. Furthermore, the versatility of the proposed method across different visual encoders is evaluated by applying it to a GroupViT-based encoder. Significant performance improvements are observed on both ViT-based CLIP and GroupViT encoders, highlighting the generalization ability of the approach. Visualization results show that attribute supervision helps accurately localize attribute-specified objects in vision-language models. Overall, this paper presents a simple yet effective approach for better aligning image and text features in vision-language models, leading to improved performance on semantic segmentation tasks.