# Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models

**Authors:** Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, Xiang Bai  
**arXiv:** [2311.06607](https://arxiv.org/abs/2311.06607)  
**Date:** 2023-11-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.06607v1/

## Key points

- Multimodal models, such as Large Multimodal Models (LMMs), have impressive capabilities in understanding general vision-language tasks
- However, these models face challenges with intricate scene understandings and narratives due to limitations in input resolution and incomplete training image-text pairs
- The Monkey method is proposed to address this issue
- Monkey allows for effective improvement in input resolution capacity up to 896 x 1344 pixels without pretraining from the start
- Monkey introduces a multi-level description generation method that provides rich information to guide models in learning contextual associations between scenes and objects
- Extensive testing across more than 16 distinct datasets shows that Monkey consistently achieves competitive performance compared to existing LMMs on tasks like Image Captioning, General VQA, and Document-oriented VQA
- Models, an interactive demo, and source code for Monkey are available on the GitHub repository (https://github.com/Yuliang-Liu/Monkey)
- Competitive performance on various vision-language tasks demonstrates the effectiveness of Monkey

## Summary

, , , , 

Multimodal models, such as Large Multimodal Models (LMMs), have shown impressive capabilities in understanding general vision-language tasks. However, these models face challenges when dealing with intricate scene understandings and narratives due to limitations in supported input resolution and the incomplete description of training image-text pairs. To address this issue, we propose a method called the Monkey method. This approach offers two main contributions: Firstly, it allows for effective improvement in the input resolution capacity up to 896 x 1344 pixels without the need for pretraining from the start. This is achieved by building upon an existing vision encoder, such as vit-BigHuge. By increasing the input resolution capacity, Monkey enables LMMs to better handle complex scenes and improve their overall performance. Secondly, Monkey introduces a multi-level description generation method that automatically provides rich information to guide models in learning contextual associations between scenes and objects. This approach enhances the model's ability to understand and interpret visual content accurately. Extensive testing across more than 16 distinct datasets demonstrates that Monkey consistently achieves competitive performance compared to existing LMMs on fundamental tasks like Image Captioning, General Visual Question Answering (VQA), and Document-oriented VQA. To facilitate further exploration and implementation of Monkey, we provide models, an interactive demo, and the source code on our GitHub repository (https://github.com/Yuliang-Liu/Monkey). In conclusion, Its effectiveness is demonstrated through competitive performance on various vision-language tasks.