Running summarizing tools on a new article
Check the paper below, then start its synthesis.
VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval
Junjie Zhou (Beijing University of Posts and Telecommunications, State Key Laboratory of Networking and Switching Technology), Zheng Liu (Hong Kong Polytechnic University, Beijing Academy of Artificial Intelligence), Shitao Xiao (Beijing Academy of Artificial Intelligence), Bo Zhao (Beijing Academy of Artificial Intelligence), Yongping Xiong (Beijing University of Posts and Telecommunications, State Key Laboratory of Networking and Switching Technology)
Abstract: Multi-modal retrieval becomes increasingly popular in practice.However, the existing retrievers are mostly text-oriented, which lack the capability to process visual information.Despite the presence of vision-language models like CLIP, the current methods are severely limited in representing the text-only and imageonly data.In this work, we present a new embedding model VISTA for universal multimodal retrieval.Our work brings forth threefold technical contributions.Firstly, we introduce a flexible architecture which extends a powerful text encoder with the image understanding capability by introducing visual token embeddings.Secondly, we develop two data generation strategies, which bring highquality composed image-text to facilitate the training of the embedding model.Thirdly, we introduce a multi-stage training algorithm, which first aligns the visual token embedding with the text encoder using massive weakly labeled data, and then develops multi-modal representation capability using the generated composed image-text data.In our experiments, VISTA achieves superior performances across a variety of multi-modal retrieval tasks in both zero-shot and supervised settings.Our model, data, and source code are available at https://github.com/FlagOpen/FlagEmbedding.