# WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
**Authors:** Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Xinyue Yang, Jiadai Sun, Yu Yang, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, Yuxiao Dong
**arXiv:** [2411.02337](https://arxiv.org/abs/2411.02337)
**Date:** 2024-11-04
**Summary page:** https://www.summarizepaper.com/en/paper/2411.02337v1/
## Key points
- Large language models (LLMs) are powerful tools for web-based tasks in the realm of autonomous agents.
- Current LLM web agents face challenges such as reliance on costly proprietary APIs and lack of decision-making capabilities.
- WebRL is a groundbreaking framework designed to train high-performance web agents using open LLMs.
- WebRL addresses critical challenges including scarcity of training tasks, sparse feedback signals, and policy distribution drift in online learning.
- Key components of WebRL include a self-evolving curriculum, robust outcome-supervised reward model (ORM), and adaptive reinforcement learning strategies.
- Application of WebRL has transformed open Llama-3.1 and GLM-4 models into proficient web agents with significantly improved success rates on the WebArena-Lite platform.
- Results show that these open models outperform established models like GPT-4-Turbo and GPT-4o while surpassing previous state-of-the-art web agents trained on open LLMs like AutoWebGLM.
- The effectiveness of WebRL highlights its role in bridging the gap between open and proprietary LLM-based web agents, advancing artificial intelligence and autonomous agent development.
## Summary
In the realm of autonomous agents, large language models (LLMs) have emerged as powerful tools for web-based tasks. However, current LLM web agents often rely on costly proprietary LLM APIs, while open LLMs lack the necessary decision-making capabilities. To address this gap, a groundbreaking framework called WebRL has been introduced.
WebRL is a self-evolving online curriculum reinforcement learning system designed to train high-performance web agents using open LLMs. It tackles three critical challenges faced in constructing LLM web agents: the scarcity of training tasks, sparse feedback signals, and policy distribution drift in online learning. This innovative framework incorporates several key components to ensure effective training.
Firstly, it features a self-evolving curriculum that generates new tasks from unsuccessful attempts. Secondly, it utilizes a robust outcome-supervised reward model (ORM) to provide feedback for learning. Lastly, adaptive reinforcement learning strategies are employed to guarantee consistent improvements over time.
Through the application of WebRL, open Llama-3.1 and GLM-4 models have been transformed into proficient web agents. Notably, on the WebArena-Lite platform,, WebRL has significantly enhanced the success rates of these models - boosting Llama-3.1-8B from 4.8% to an impressive 42.4%, and elevating GLM-4-9B from 6.1% to 43%. These results demonstrate that these open models surpass the performance of established models like GPT-4-Turbo and GPT-4o while outperforming previous state-of-the-art web agents trained on open LLMs such as AutoWebGLM.
The findings underscore the effectiveness of WebRL in bridging the divide between open and proprietary LLM-based web agents. By paving the way for more accessible and powerful autonomous web interaction systems, WebRL represents a significant advancement in the field of artificial intelligence and autonomous agent development.