# OmChat: A Recipe to Train Multimodal Language Models with Strong Long Context and Video Understanding

**Authors:** Tiancheng Zhao, Qianqian Zhang, Kyusong Lee, Peng Liu, Lu Zhang, Chunxin Fang, Jiajia Liao, Kelei Jiang, Yibo Ma, Ruochen Xu  
**arXiv:** [2407.04923](https://arxiv.org/abs/2407.04923)  
**Date:** 2024-07-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.04923v1/

## Key points

- OmChat is a cutting-edge model designed for handling long multimodal contexts and understanding video data
- The model efficiently manages sequences of images and video frames spanning significant temporal lengths
- Utilizes an active progressive multimodal pretraining strategy to enhance processing capabilities
- Learns from high-quality data points during training for robust performance across tasks
- Supports a context length of up to 512K tokens, ideal for tasks involving multiple images and videos
- Outperforms other open-source models in benchmarks for complex visual data interpretation
- Achieves competitive performance on single-image benchmarks, often surpassing larger size models
- Key factors contributing to OmChat's success include support for higher image resolutions, active progressive pretraining strategy, and incorporation of high-quality supervised fine-tuning datasets
- These elements collectively enhance efficiency, adaptability, and overall performance in visual understanding tasks

## Summary

OmChat is a cutting-edge model specifically designed to excel in handling long multimodal contexts and understanding video data. With the increasing importance of processing both textual and visual inputs in AI systems, OmChat stands out for its ability to efficiently manage sequences of images and video frames that span significant temporal lengths. The model's architecture standardizes how different visual inputs are processed, making it more efficient and adaptable. Utilizing an active progressive multimodal pretraining strategy, OmChat gradually scales its capacity for processing long contexts, enhancing its overall capabilities. By selectively utilizing high-quality data during training, OmChat learns from the most relevant and informative data points, ensuring robust performance across various tasks. With support for a context length of up to 512K tokens, OmChat is well-suited for tasks involving multiple images and videos. In benchmarks for these tasks, OmChat consistently outperforms all other open-source models, showcasing its superior ability to manage and interpret complex visual data. Additionally, the model achieves competitive performance on single-image benchmarks, often surpassing larger size models. To further evaluate OmChat's capabilities, a benchmark dataset named Temporal Visual Needle in a Haystack was proposed to assess the model's ability to comprehend temporal visual details within videos. Several key factors contribute to OmChat's success: support for higher image resolutions  the active progressive pretraining strategy  and the incorporation of high-quality supervised fine-tuning datasets  These elements collectively enhance OmChat's efficiency  adaptability  and overall performance in visual understanding tasks. The comprehensive overview provided in this paper details OmChat's architecture  training methodology  and performance across various benchmarks. The findings underscore the significance of higher image resolutions  progressive multimodal pretraining  and high-quality data selection in achieving state-of-the-art performance in multimodal large language models. Overall, OmChat represents a significant advancement in handling long contexts and video understanding tasks with promising results across diverse applications.