# FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression

**Authors:** Yuke Zhu, Chi Xie, Shuang Liang, Bo Zheng, Sheng Guo  
**arXiv:** [2411.14228](https://arxiv.org/abs/2411.14228)  
**Date:** 2024-11-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2411.14228v1/

## Key points

- High-resolution image input is crucial for enhancing model capabilities in Multi-modal Large Language Models (MLLMs)
- Increased visual tokens input leads to significant computational costs
- Research focuses on developing visual token compression methods to improve efficiency without compromising performance
- FocusLLaVA is a promising solution that removes visual redundancy and enhances efficiency and performance
- FocusLLaVA incorporates vision-guided sampler and text-guided sampler modules for efficient visual token compression
- Textual guidance becomes more accurate and stable as layers go deeper in LLMs
- Placing the text-guided sampler in middle layers is crucial for optimal performance
- FocusLLaVA outperforms state-of-the-art MLLMs in terms of efficiency and performance by utilizing both visual and textual information effectively

## Summary

In recent advancements in Multi-modal Large Language Models (MLLMs), it has been established that high-resolution image input is essential for enhancing model capabilities, particularly for fine-grained tasks. However, this also leads to a quadratic increase in the number of visual tokens input into LLMs, resulting in significant computational costs. To address this challenge, current research has focused on developing visual token compression methods to improve efficiency without compromising performance. One promising solution is FocusLLaVA, which aims to remove visual redundancy and simultaneously enhance both efficiency and performance. This approach incorporates two key modules: a vision-guided sampler and a text-guided sampler. The vision-guided sampler focuses on areas of high information density within images, such as text, patterns, and people. On the other hand, the text-guided sampler emphasizes regions directly related to user queries or instructions. By leveraging these two modules in a coarse-to-fine visual token compression method, FocusLLaVA achieves improvements in both efficiency and performance. Analysis of importance maps across different layers of LLMs reveals that textual guidance becomes more accurate and stable as the layers go deeper. This highlights the progressive nature of LLM's understanding of the relationship between image and text information. Placing the text-guided sampler in middle layers proves crucial for optimal performance. Extensive experiments conducted on various multimodal benchmarks demonstrate that FocusLLaVA outperforms state-of-the-art MLLMs in terms of efficiency and performance by effectively utilizing both visual and textual information as guidance mechanisms. Furthermore, related work highlights the evolution from early models like BLIP2 to more recent approaches like LLaVA 1.5 that have optimized image-text alignment processes within MLLMs. The development of FocusLLaVA represents an important step towards achieving efficient and effective visual token compression within multimodal large language models.