# DroidBot-GPT: GPT-powered UI Automation for Android

**Authors:** Hao Wen, Hongming Wang, Jiaxuan Liu, Yuanchun Li  
**arXiv:** [2304.07061](https://arxiv.org/abs/2304.07061)  
**Date:** 2023-04-14  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2304.07061v5/

## Key points

- Authors introduce DroidBot-GPT, a tool powered by GPT for automating interactions with Android apps
- DroidBot-GPT interprets natural language descriptions of tasks to navigate apps and accomplish them
- The tool prompts the Language Model (LLM) to select appropriate actions based on GUI state information and available actions on the smartphone screen
- Evaluation using a dataset shows DroidBot-GPT completes 39.39% of tasks with an average partial completion progress of approximately 66.76%
- Method requires no modifications to target application or LLM, making it fully unsupervised
- Research highlights potential for enhancing automation performance through improved app development paradigms or custom model training strategies

## Summary

, , , , 


In their paper titled "DroidBot-GPT: GPT-powered UI Automation for Android," authors Hao Wen, Hongming Wang, Jiaxuan Liu, and Yuanchun Li introduce DroidBot-GPT, a novel tool that leverages  to automate interactions with . The tool operates by interpreting natural language descriptions of desired tasks and generating corresponding actions to navigate the app and accomplish the task. By translating the app's graphical user interface (GUI) state information and available actions on the smartphone screen into natural language prompts, DroidBot-GPT prompts the LLM to select appropriate actions. Since the LLM is trained on extensive data encompassing how-to manuals of various software applications, it can make informed decisions based on provided information. The authors evaluate DroidBot-GPT using a self-curated dataset comprising 33 tasks sourced from 17 Android applications across 10 categories. Results indicate that DroidBot-GPT successfully completes 39.39% of tasks, with an average partial completion progress of approximately 66.76%. Notably, this method requires no modifications to either the target application or the LLM, rendering it fully unsupervised. The authors posit that there exists significant potential to enhance automation performance through improved app development paradigms or custom model training strategies. This research underscores the efficacy of employing advanced language models like GPT for automating complex interactions within mobile applications. By bridging natural language descriptions with actionable commands, DroidBot-GPT showcases promising capabilities in streamlining app navigation and task execution processes. The findings suggest avenues for further refinement and optimization in  through continued advancements in both software development practices and machine learning model training methodologies.