# Apollo: An Exploration of Video Understanding in Large Multimodal Models

**Authors:** Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, Serena Yeung-Levy, Xide Xia  
**arXiv:** [2412.10360](https://arxiv.org/abs/2412.10360)  
**Date:** 2024-12-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2412.10360v1/

## Key points

- Rapid integration of video perception capabilities into Large Multimodal Models (LMMs)
- Poor understanding of underlying mechanisms driving video understanding in LMMs
- High computational cost associated with training and evaluating video-LMMs
- Comprehensive study conducted to uncover key factors driving video understanding in LMMs
- Scaling Consistency identified as a key factor influencing computational requirements
- Importance of exploring various video-specific aspects in designing video-LMMs, such as fps sampling, vision encoders, and data composition
- Introduction of Apollo as a state-of-the-art family of LMMs achieving superior performance across different model sizes
- Need for specialized strategies when designing video-LMMs due to unique challenges
- Aim to democratize video-LMM research and accelerate advancements in the field by providing guidelines and resources for future research

## Summary

In recent years, there has been a rapid integration of video perception capabilities into Large Multimodal Models (LMMs). However, the underlying mechanisms driving video understanding in these models remain poorly understood. This has led to design decisions being made without proper justification or analysis. The high computational cost associated with training and evaluating such models, coupled with limited open research in this area, has hindered the development of video-LMMs. To address these challenges, a comprehensive study was conducted to uncover what effectively drives video understanding in LMMs. The study began by critically examining the primary contributors to the high computational requirements of video-LMM research and identified Scaling Consistency as a key factor. This concept suggests that design and training decisions made on smaller models and datasets can effectively transfer to larger models up to a critical size. Building on these insights, the study explored various video-specific aspects of video-LMMs including video sampling techniques, architectures, data composition, training schedules, and more. For example, it was demonstrated that fps sampling during training is preferable to uniform frame sampling and certain vision encoders are better suited for video representation. Guided by these findings, Apollo was introduced as a state-of-the-art family of LMMs that achieve superior performance across different model sizes. Apollo-3B outperformed most existing $7$B models with an impressive score on LongVideoBench. Additionally, Apollo-7B surpassed all 7B LMMs on MLVU and Video-MME benchmarks. The study also highlighted the importance of systematically exploring the design space for image-based LMMs and emphasized the need for specialized strategies when designing video-LMMs due to their unique challenges. By addressing these gaps in research and providing valuable insights into key aspects of video-LMM design,this work aims to democratize video-LMM research and accelerate advancements in the field. In conclusion, the study provides guidelines and resources for future research in developing efficient and effective video-LMMs. The findings suggest that careful design and training strategies can lead to superior performance without necessarily requiring larger model sizes. Overall, this work contributes to advancing the development of scalable solutions for video understanding within Large Multimodal Models.