OmChat: A Recipe to Train Multimodal Language Models with Strong Long Context and Video Understanding

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Close Not relevant

   VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding

   Yue Fan, Xiaojian Ma, Rujie Wu et al. · 2024 · Computer Vision and Pattern Recognition · AI summary available

   Abstract

   We explore how reconciling several foundation models (large language models and vision-language models) with a novel unified memory mechanism could tackle the challenging video understanding problem, especially capturing the long-term temporal relations in lengthy videos. In particular, the proposed multimodal agent VideoAgent: 1) constructs a structured memory to store both the generic temporal event descriptions and object-centric tracking states of the video; 2) given an input task query, it employs tools including video segment localization and object memory querying along with other visual foundation models to interactively solve the task, utilizing the zero-shot tool-use ability of LLM…

2. Close Not relevant

   MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

   Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier et al. · 2024 · Computer Vision and Pattern Recognition · AI summary available

   Abstract

   In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons. For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published pre-training results. Further, we show that the image encoder t…

3. Close Not relevant

   MoVA: Adapting Mixture of Vision Experts to Multimodal Context

   Zhuofan Zong, Bingqi Ma, Dazhong Shen et al. · 2024 · Computer Vision and Pattern Recognition · AI summary available

   Abstract

   As the key component in multimodal large language models (MLLMs), the ability of the visual encoder greatly affects MLLM's understanding on diverse image content. Although some large-scale pretrained vision encoders such as vision encoders in CLIP and DINOv2 have brought promising performance, we found that there is still no single vision encoder that can dominate various image content understanding, e.g., the CLIP vision encoder leads to outstanding results on general image understanding but poor performance on document or chart content. To alleviate the bias of CLIP vision encoder, we first delve into the inherent behavior of different pre-trained vision encoders and then propose the MoVA,…

4. Close Not relevant

   Generative Pretraining in Multimodality

   Quan Sun, Qiying Yu, Yufeng Cui et al. · 2023 · Computer Vision and Pattern Recognition · AI summary available

   Abstract

   We present Emu, a Transformer-based multimodal foundation model, which can seamlessly generate images and texts in multimodal context. This omnivore model can take in any single-modality or multimodal data input indiscriminately (e.g., interleaved image, text and video) through a one-model-for-all autoregressive training process. First, visual signals are encoded into embeddings, and together with text tokens form an interleaved input sequence. Emu is then end-to-end trained with a unified objective of classifying the next text token or regressing the next visual embedding in the multimodal sequence. This versatile multimodality empowers the exploration of diverse pretraining data sources at…

5. Close Not relevant

   Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models

   Zhang Li, Biao Yang, Qiang Liu et al. · 2023 · Computer Vision and Pattern Recognition · AI summary available

   Abstract

   Large Multimodal Models have demonstrated impressive capabilities in understanding general vision-language tasks. However, due to the limitation of supported input resolution (e.g., 448 x 448) as well as the inexhaustive description of the training image-text pair, these models often encounter challenges when dealing with intricate scene understandings and narratives. Here we address the problem by proposing the Monkey. Our contributions are two-fold: 1) without pretraining from the start, our method can be built upon an existing vision encoder (e.g., vit-BigHuge) to effectively improve the input resolution capacity up to 896 x 1344 pixels; 2) we propose a multi-level description generation …

6. Close Not relevant

   Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

   Chaoyou Fu, Yuhan Dai, Yondong Luo et al. · 2024 · Computer Vision and Pattern Recognition · AI summary available

   Abstract

   In the quest for artificial general intelligence, Multi-modal Large Language Models (MLLMs) have emerged as a focal point in recent advancements. However, the predominant focus remains on developing their capabilities in static image understanding. The potential of MLLMs in processing sequential visual data is still insufficiently explored, highlighting the absence of a comprehensive, high-quality assessment of their performance. In this paper, we introduce Video-MME, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of MLLMs in Video analysis. Our work distinguishes from existing benchmarks through four key features: 1) Diversity in video types, spanning 6 primary visual domain…

7. Close Not relevant

   Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Models

   Rui Qian, Yeqing Li, Zheng Xu et al. · 2022 · Computer Vision and Pattern Recognition · AI summary available

   Abstract

   Utilizing vision and language models (VLMs) pre-trained on large-scale image-text pairs is becoming a promising paradigm for open-vocabulary visual recognition. In this work, we extend this paradigm by leveraging motion and audio that naturally exist in video. We present \textbf{MOV}, a simple yet effective method for \textbf{M}ultimodal \textbf{O}pen-\textbf{V}ocabulary video classification. In MOV, we directly use the vision encoder from pre-trained VLMs with minimal modifications to encode video, optical flow and audio spectrogram. We design a cross-modal fusion mechanism to aggregate complimentary multimodal information. Experiments on Kinetics-700 and VGGSound show that introducing flow…

8. Close Not relevant

   Meta-Transformer: A Unified Framework for Multimodal Learning

   Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang et al. · 2023 · Computer Vision and Pattern Recognition · AI summary available

   Abstract

   Multimodal learning aims to build models that can process and relate information from multiple modalities. Despite years of development in this field, it still remains challenging to design a unified network for processing various modalities ($\textit{e.g.}$ natural language, 2D images, 3D point clouds, audio, video, time series, tabular data) due to the inherent gaps among them. In this work, we propose a framework, named Meta-Transformer, that leverages a $\textbf{frozen}$ encoder to perform multimodal perception without any paired multimodal training data. In Meta-Transformer, the raw input data from various modalities are mapped into a shared token space, allowing a subsequent encoder wi…