# OmniParser for Pure Vision Based GUI Agent

**Authors:** Yadong Lu, Jianwei Yang, Yelong Shen, Ahmed Awadallah  
**arXiv:** [2408.00203](https://arxiv.org/abs/2408.00203)  
**Date:** 2024-08-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2408.00203v1/

## Key points

- OMNIPARSER is a vision-only approach designed to parse UI screenshots into structured elements
- Consists of two fine-tuned models: icon detection model and functional description model
- Curated datasets for interactable region detection and icon functional description training
- Performance enhancement of GPT-4V on ScreenSpot benchmarks with parsed results from OMNIPARSER
- Outperforms GPT-4V agents relying on HTML-extracted information on Mind2Web benchmark and surpasses GPT-4V augmented with specialized Android icon detection model on AITW benchmark
- Goal is to provide a versatile tool for parsing UI screens across PC and mobile platforms without additional information like HTML or view hierarchy in Android
- Addresses limitations in creating widely usable agents across multiple platforms and applications
- Vision-based screen parsing technique bridges understanding of basic UI elements and grounding actions in various operating systems and applications
- Reliable vision-based screen parsing method crucial for enhancing robustness of agentic workflows in diverse user tasks
- OMNIPARSER extracts information from UI screenshots, providing structured bounding boxes and labels to improve GPT-4V's performance in action prediction across different user tasks

## Summary

, , , , 


In this report, we introduce OMNIPARSER, a comprehensive vision-only approach designed to parse user interface (UI) screenshots into structured elements. The system consists of two fine-tuned models: an icon detection model and a functional description model. To train these models, we curated an interactable region detection dataset using popular webpages and an icon functional description dataset. By leveraging the parsed results from OMNIPARSER, the performance of GPT-4V is significantly enhanced on ScreenSpot benchmarks. Notably, OMNIPARSER outperforms GPT-4V agents that rely on HTML-extracted information on the Mind2Web benchmark and surpasses GPT-4V augmented with a specialized Android icon detection model on the AITW benchmark. Our goal with OMNIPARSER is to provide a versatile and user-friendly tool capable of parsing UI screens across both PC and mobile platforms without the need for additional information such as HTML or view hierarchy in Android. This approach addresses the current limitations in creating widely usable agents that can operate seamlessly across multiple platforms and applications. While previous works have focused on specific applications or platforms, our vision-based screen parsing technique aims to bridge the gap between understanding basic UI elements and accurately grounding actions in various operating systems and applications. We argue that existing pure vision-based screen parsing techniques are inadequate, leading to an underestimation of the capabilities of models like GPT-4V. A reliable vision-based screen parsing method is crucial for enhancing the robustness of agentic workflows in diverse user tasks. OMNIPARSER serves as a general solution for extracting information from UI screenshots, providing structured bounding boxes and labels that improve GPT-4V's performance in action prediction across different user tasks. Overall, our contributions include curating an interactable region detection dataset based on bounding boxes extracted from DOM trees of popular webpages, developing specialized models for icon detection and functional description, and demonstrating significant performance improvements on benchmark tests. We envision OMNIPARSER as a valuable tool for advancing research in agent systems operating on UI interfaces and facilitating seamless interactions across multiple platforms and applications.