# Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

**Authors:** Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, Zhe Gan  
**arXiv:** [2404.05719](https://arxiv.org/abs/2404.05719)  
**Date:** 2024-04-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.05719v1/

## Key points

- Ferret-UI is a novel multimodal large language model (MLLM) designed to enhance comprehension and interaction with user interface (UI) screens
- Equipped with referring, grounding, and reasoning capabilities tailored for mobile UI screens
- Incorporates "any resolution" to magnify details and leverage enhanced visual features
- Training data includes elementary UI tasks such as icon recognition, finding text, and widget listing formatted with region annotations
- Dataset for advanced tasks compiled to enhance reasoning ability
- Outperforms most open-source UI MLLMs and surpasses GPT-4V on all elementary UI tasks

## Summary

, , , , 

In their paper titled "Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs," authors Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan introduce Ferret-UI as a novel multimodal large language model (MLLM) specifically designed to enhance the comprehension and interaction capabilities with user interface (UI) screens. Existing general-domain MLLMs have shown remarkable progress in various tasks; however, they often struggle when it comes to effectively understanding and engaging with UI screens. To address this limitation, Ferret-UI is equipped with referring, grounding, and reasoning capabilities tailored for mobile UI screens. Recognizing that UI screens typically feature an elongated aspect ratio and contain smaller objects of interest like icons and text compared to natural images, the researchers incorporate "any resolution" into Ferret-UI to magnify details and leverage enhanced visual features. Each screen is divided into two sub-images based on the original aspect ratio - horizontal division for portrait screens and vertical division for landscape screens. These sub-images are encoded separately before being processed by the MLLMs. The training data for Ferret-UI is meticulously gathered from a wide range of elementary UI tasks such as icon recognition, finding text, and widget listing. These samples are formatted with region annotations to facilitate precise referring and grounding during instruction-following processes. Additionally, a dataset for advanced tasks including detailed descriptions, perception/interaction conversations, and function inference is compiled to enhance the model's reasoning ability. After training on these curated datasets, demonstrates exceptional comprehension of  and the capability to execute open-ended instructions effectively. To evaluate its performance comprehensively across all mentioned tasks, the researchers establish a  where Ferret-UI not only outperforms most open-source UI MLLMs but also surpasses GPT-4V on all elementary UI tasks. This innovative approach towards grounded mobile UI understanding showcases significant advancements in  tailored specifically for interacting with user interface screens efficiently.