VindLU: A Recipe for Effective Video-and-Language Pretraining

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Builds on this paper Cites this paper Not relevant

   VidLA: Video-Language Alignment at Scale

   Mamshad Nayeem Rizve, Fan Fei, Jayakrishnan Unnikrishnan et al. · 2024 · Computer Vision and Pattern Recognition · cited 13 times · AI summary available

   VidLA uses a simple two-tower architecture and temporally hierarchical data tokens for large-scale video-language alignment; it cites VindLU.

   Abstract

   In this paper, we propose VidLA, an approach for video-language alignment at scale. There are two major limitations of previous video-language alignment approaches. First, they do not capture both short-range and long-range temporal dependencies and typically employ complex hierarchical deep network architectures that are hard to integrate with existing pretrained image-text foundation models. To effectively address this limitation, we instead keep the network architecture simple and use a set of data tokens that operate at different temporal resolutions in a hierarchical manner, accounting for the temporally hierarchical nature of videos. By employing a simple two-tower architecture, we are…

2. Very close Broader context Cites this paper Uses this paper in depth Not relevant

   ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models

   Ilker Kesen, Andrea Pedrotti, Mustafa Doğan et al. · 2023 · International Conference on Learning Representations · cited 31 times · Not summarized yet

   ViLMA evaluates fine-grained linguistic and temporal grounding in video-language models with controlled counterfactuals; it cites VindLU.

   Abstract

   With the ever-increasing popularity of pretrained Video-Language Models (VidLMs), there is a pressing need to develop robust evaluation methodologies that delve deeper into their visio-linguistic capabilities. To address this challenge, we present ViLMA (Video Language Model Assessment), a task-agnostic benchmark that places the assessment of fine-grained capabilities of these models on a firm footing. Task-based evaluations, while valuable, fail to capture the complexities and specific temporal aspects of moving images that VidLMs need to process. Through carefully curated counterfactuals, ViLMA offers a controlled evaluation suite that sheds light on the true potential of these models, as …

3. Close Another approach to the problem Cites this paper Not relevant

   VLAB: Enhancing Video Language Pretraining by Feature Adapting and Blending

   Xingjian He, Sihan Chen, Fan Ma et al. · 2023 · IEEE transactions on multimedia · cited 32 times · Not summarized yet

   VLAB transfers CLIP representations to video-language pretraining through feature adapting and blending; it cites VindLU.

   Abstract

   Large-scale image-text contrastive pre-training models, such as CLIP, have been demonstrated to effectively learn high-quality multimodal representations. However, there is limited research on learning video-text representations for general video multimodal tasks based on these powerful features. Towards this goal, we propose a novel video-text pre-training method dubbed VLAB: Video Language pre-training by feature Adapting and Blending, which transfers CLIP representations to video pre-training tasks and develops unified video multimodal models for a wide range of video-text tasks. Specifically, VLAB is founded on two key strategies: feature adapting and feature blending. In the former, we …

4. Close Another approach to the problem Not relevant

   VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding

   Hu Xu, Gargi Ghosh, Po-Yao (Bernie) Huang et al. · 2021 · Findings · cited 160 times · Not summarized yet

   VLM introduces task-agnostic multimodal masking schemes for video understanding, rather than VindLU’s empirical recipe across pretraining design choices.

   Abstract

   We present a simplified, task-agnostic multi-modal pre-training approach that can accept either video or text input, or both for a variety of end tasks. Existing pre-training are task-specific by adopting either a single cross-modal encoder that requires both modalities, limiting their use for retrieval-style end tasks or more complex multitask learning with two unimodal encoders, limiting early cross-modal fusion. We instead introduce new pretraining masking schemes that better mix across modalities (e.g. by forcing masks for text to predict the closest video embeddings) while also maintaining separability (e.g. unimodal predictions are sometimes required, without using all the input). Expe…

5. Close Same question Cited by this paper Not relevant

   CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment

   Hong-Wei Xue, Yu-Chong Sun, Bei Liu et al. · 2022 · International Conference on Learning Representations · cited 122 times · Not summarized yet

   CLIP-ViP investigates factors limiting CLIP post-pretraining for video-language tasks and ways to address them; VindLU cites it.

   Abstract

   The pre-trained image-text models, like CLIP, have demonstrated the strong power of vision-language representation learned from a large scale of web-collected image-text data. In light of the well-learned visual features, some existing works transfer image representation to video domain and achieve good results. However, how to utilize image-language pre-trained model (e.g., CLIP) for video-language pre-training (post-pretraining) is still under explored. In this paper, we investigate two questions: 1) what are the factors hindering post-pretraining CLIP to further improve the performance on video-language tasks? and 2) how to mitigate the impact of these factors? Through a series of compara…

6. Close Another approach to the problem Not relevant

   LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling

   Dongsheng Chen, Chaofan Tao, Lu Hou et al. · 2022 · Conference on Empirical Methods in Natural Language Processing · cited 20 times · Not summarized yet

   LiteVL adapts BLIP to video with temporal attention modules and avoids heavy pretraining, unlike VindLU’s pretraining recipe.

   Abstract

   Recent large-scale video-language pre-trained models have shown appealing performance on various downstream tasks. However, the pre-training process is computationally expensive due to the requirement of millions of video-text pairs and the redundant data structure of each video. To mitigate these problems, we propose LiteVL, which adapts a pre-trained image-language model BLIP into a video-text model directly on downstream tasks, without heavy pre-training. To enhance the temporal modeling lacking in the image-language model, we propose to add temporal attention modules in the image encoder of BLIP with dynamic temporal scaling. Besides the model-wise adaptation, we also propose a non-param…

7. Close Groundwork for this paper Cited by this paper Not relevant

   VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling

   Tsu-Jui Fu, Lin-Jie Li, Zhe Gan et al. · 2021 · arXiv.org · preprint, not peer reviewed · cited 258 times · Not summarized yet

   VIOLET uses end-to-end video-language training with masked visual-token modeling, a prior approach VindLU cites.

   Abstract

   A great challenge in video-language (VidL) modeling lies in the disconnection between fixed video representations extracted from image/video understanding models and downstream VidL data. Recent studies try to mitigate this disconnection via end-to-end training. To make it computationally feasible, prior works tend to"imagify"video inputs, i.e., a handful of sparsely sampled frames are fed into a 2D CNN, followed by a simple mean-pooling or concatenation to obtain the overall video representations. Although achieving promising results, such simple approaches may lose temporal information that is essential for performing downstream VidL tasks. In this work, we present VIOLET, a fully end-to-e…

8. Close Same method, another problem Not relevant

   Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning

   Yu-Chong Sun, Hong-Wei Xue, Rui-Hua Song et al. · 2022 · Neural Information Processing Systems · cited 94 times · Not summarized yet

   LF-VILA applies video-language pretraining to long-form videos, focusing on long-range relationships and temporal contrastive learning.

   Abstract

   Large-scale video-language pre-training has shown significant improvement in video-language understanding tasks. Previous studies of video-language pretraining mainly focus on short-form videos (i.e., within 30 seconds) and sentences, leaving long-form video-language pre-training rarely explored. Directly learning representation from long-form videos and language may benefit many long-form video-language understanding tasks. However, it is challenging due to the difficulty of modeling long-range relationships and the heavy computational burden caused by more frames. In this paper, we introduce a Long-Form VIdeo-LAnguage pre-training model (LF-VILA) and train it on a large-scale long-form vid…

Prior work it builds on

1. Close Groundwork for this paper Used in depth by this paper A method it uses Not relevant

   LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling

   Linjie Li, Zhe Gan, Kevin Lin et al. · 2022 · Computer Vision and Pattern Recognition · cited 105 times · Not summarized yet

   LAVENDER unifies video-language pretraining and downstream tasks through masked language modeling, a prior framework VindLU cites.

   Abstract

   Unified vision-language frameworks have greatly advanced in recent years, most of which adopt an encoder-decoder architecture to unify image-text tasks as sequence-to-sequence generation. However, existing video-language (VidL) models still require task-specific designs in model architecture and training objectives for each task. In this work, we explore a unified VidL framework LAVENDER, where Masked Language Modeling [13] (MLM) is used as the common interface for all pre-training and downstream tasks. Such unification leads to a simplified model architecture, where only a lightweight MLM head, instead of a decoder with much more parameters, is needed on top of the multimodal encoder. Surpr…

2. Close Groundwork for this paper A method it uses Not relevant

   ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

   Jiasen Lu, Dhruv Batra, Devi Parikh et al. · 2019 · Neural Information Processing Systems · cited 4,814 times · Not summarized yet

   ViLBERT establishes task-agnostic joint visual-language representations using two streams and co-attentional layers, and VindLU cites it.

   Abstract

   We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, pro-cessing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -- visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval -- by making only minor additions to the base archit…

3. Close Groundwork for this paper Used in depth by this paper A method it uses Not relevant

   Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling

   Jie Lei, Linjie Li, Luowei Zhou et al. · 2021 · Computer Vision and Pattern Recognition · cited 844 times · Not summarized yet

   CLIPBERT proposes end-to-end video-language learning with sparse frame sampling, an earlier approach VindLU cites.

   Abstract

   The canonical approach to video-and-language learning (e.g., video question answering) dictates a neural model to learn from offline-extracted dense video features from vision models and text features from language models. These feature extractors are trained independently and usually on tasks different from the target domains, rendering these fixed features sub-optimal for downstream tasks. Moreover, due to the high computational overload of dense video features, it is often difficult (or infeasible) to plug feature extractors directly into existing approaches for easy finetuning. To provide a remedy to this dilemma, we propose a generic framework CLIPBERT that enables affordable endto-end …

4. Related Same method, another problem Used in depth by this paper A method it uses Not relevant

   Zero-Shot Video Question Answering via Frozen Bidirectional Language Models

   Antoine Yang, Antoine Miech, Josef Sivic et al. · 2022 · Neural Information Processing Systems · cited 311 times · Not summarized yet

   This work uses frozen bidirectional language models for zero-shot video question answering, a downstream task VindLU also evaluates.

   Abstract

   Video question answering (VideoQA) is a complex task that requires diverse multi-modal data for training. Manual annotation of question and answers for videos, however, is tedious and prohibits scalability. To tackle this problem, recent methods consider zero-shot settings with no manual annotation of visual question-answer. In particular, a promising approach adapts frozen autoregressive language models pretrained on Web-scale text-only data to multi-modal inputs. In contrast, we here build on frozen bidirectional language models (BiLM) and show that such an approach provides a stronger and cheaper alternative for zero-shot VideoQA. In particular, (i) we combine visual inputs with the froze…

5. Close Groundwork for this paper Not relevant

   All in One: Exploring Unified Video-Language Pre-Training

   Jin-Peng Wang, Yixiao Ge, Rui Yan et al. · 2022 · Computer Vision and Pattern Recognition · cited 257 times · Not summarized yet

   All in One proposes a unified Transformer for raw video and text inputs, an earlier video-language pretraining model VindLU cites.

   Abstract

   Mainstream Video-Language Pre-training (VLP) models [10, 26, 64] consist of three parts, a video encoder, a text encoder, and a video-text fusion Transformer. They pursue better performance via utilizing heavier unimodal encoders or multimodal fusion Transformers, resulting in increased parameters with lower efficiency in downstream tasks. In this work, we for the first time introduce an end-to-end VLP model, namely all-in-one Transformer, that embeds raw video and textual signals into joint representations using a unified backbone architecture. We argue that the unique temporal information of video data turns out to be a key barrier hindering the design of a modality-agnostic Transformer. T…

Papers that cite it

1. Close Builds on this paper Uses this paper in depth Not relevant

   InternVideo2: Scaling Foundation Models for Multimodal Video Understanding

   Yi Wang, Kun-Chang Li, Xin-Hao Li et al. · 2024 · European Conference on Computer Vision · cited 466 times · Not summarized yet

   InternVideo2 scales video foundation models with masked modeling, cross-modal contrastive learning, and next-token prediction; it cites VindLU.

   Abstract

   We introduce InternVideo2, a new family of video foundation models (ViFM) that achieve the state-of-the-art results in video recognition, video-text tasks, and video-centric dialogue. Our core design is a progressive training approach that unifies the masked video modeling, crossmodal contrastive learning, and next token prediction, scaling up the video encoder size to 6B parameters. At the data level, we prioritize spatiotemporal consistency by semantically segmenting videos and generating video-audio-speech captions. This improves the alignment between video and text. Through extensive experiments, we validate our designs and demonstrate superior performance on over 60 video and audio task…

2. Close Builds on this paper Uses this paper in depth Not relevant

   Unmasked Teacher: Towards Training-Efficient Video Foundation Models

   Kun-Chang Li, Ya-Li Wang, Yi-Zhuo Li et al. · 2023 · IEEE International Conference on Computer Vision · cited 297 times · Not summarized yet

   Unmasked Teacher trains temporal-sensitive video foundation models by aligning unmasked tokens with an image foundation model; it cites VindLU.

   Abstract

   Video Foundation Models (VFMs) have received limited exploration due to high computational costs and data scarcity. Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video domain. Although VideoMAE has trained a robust ViT from limited data, its low-level reconstruction poses convergence difficulties and conflicts with high-level cross-modal alignment. This paper proposes a training-efficient method for temporal-sensitive VFMs that integrates the benefits of existing methods. To increase data efficiency, we mask out most of the low-semantics video tokens, but selectively align the unmasked tokens with IFM, which serves as the UnMasked Teacher …

3. Close Same method, another problem Uses this paper in depth Not relevant

   Videoprompter: an ensemble of foundational models for zero-shot video understanding

   Adeel Yousaf, Muzammal Naseer, Salman H. Khan et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 4 times · Not summarized yet

   Videoprompter combines pretrained vision-language and generative models for zero-shot video understanding; it cites VindLU.

   Abstract

   Vision-language models (VLMs) classify the query video by calculating a similarity score between the visual features and text-based class label representations. Recently, large language models (LLMs) have been used to enrich the text-based class labels by enhancing the descriptiveness of the class names. However, these improvements are restricted to the text-based classifier only, and the query visual features are not considered. In this paper, we propose a framework which combines pre-trained discriminative VLMs with pre-trained generative video-to-text and text-to-text models. We introduce two key modifications to the standard zero-shot setting. First, we propose language-guided visual fea…

4. Close Same method, another problem Uses this paper in depth Not relevant

   CliPPER: Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition

   F. Stilz, V. Srivastav, N. Navab et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 3 times · Not summarized yet

   CliPPER pretrains video-language models for fine-grained temporal recognition in long surgical videos; it cites VindLU.

   Abstract

   Video-language foundation models have proven to be highly effective in zero-shot applications across a wide range of tasks. A particularly challenging area is the intraoperative surgical procedure domain, where labeled data is scarce, and precise temporal understanding is often required for complex downstream tasks. To address this challenge, we introduce CliPPER (Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition), a novel video-language pretraining framework trained on surgical lecture videos. Our method is designed for fine-grained temporal video-text recognition and introduces several novel pretraining strategies to improve multimo…

5. Close Same method, another problem Not relevant

   VicTR: Video-conditioned Text Representations for Activity Recognition

   Kumara Kahatapitiya, Anurag Arnab, Arsha Nagrani et al. · 2023 · Computer Vision and Pattern Recognition · cited 47 times · Not summarized yet

   VicTR augments text representations with video information for activity recognition, rather than focusing on a general pretraining recipe; it cites VindLU.

   Abstract

   Vision-Language models (VLMs) have excelled in the image-domain- especially in zero-shot settings- thanks to the availability of vast pretraining data (i.e., paired image-text samples). However for videos, such paired data is not as abundant. Therefore, video- VLMs are usually designed by adapting pretrained image- VLMs to the video-domain, instead of training from scratch. All such recipes rely on aug-menting visual embeddings with temporal information (i.e., image -+ video), often keeping text embeddings unchanged or even being discarded. In this paper, we argue the contrary, that better video- VLMs can be designed by focusing more on augmenting text, rather than visual information. More s…