# ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents

**Authors:** Hao Zhang, Mingjie Liu, Shaokun Zhang, Songyang Han, Jian Hu, Zhenghui Jin, Yuchi Zhang, Shizhe Diao, Ximing Lu, Binfeng Xu, Zhiding Yu, Jan Kautz, Yi Dong  
**arXiv:** [2603.18815](https://arxiv.org/abs/2603.18815)  
**Date:** 2026-03-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2603.18815v1/

## Key points

- ProRL Agent is a revolutionary infrastructure addressing the main limitation of current agentic RL training frameworks
- Focuses on multi-turn agentic rollout and decouples it from the trainer through a unified HTTP interface for scalable agent RL training
- Innovative token-in/token-out trajectory communication eliminates re-tokenization, streamlining the training process
- Offers standardized and extensible sandbox environments supporting various agentic tasks in rootless HPC settings
- Supports various reinforcement learning algorithms like PPO and GRPO for adaptability to different training scenarios
- Provides REST API for rollout requests and detailed evaluation metrics like rewards and trajectories to simplify RL training process
- Enhances long-horizon behavior improvement in multi-turn LLM agents with its "rollout-as-a-service" philosophy
- Integrated into NVIDIA NeMo Gym, making it a cutting-edge tool for researchers and developers working on complex interactive tasks involving multi-turn LLM agents

## Summary

ProRL Agent is a revolutionary infrastructure that addresses the main limitation of current agentic RL training frameworks. It focuses on multi-turn agentic rollout and decouples it from the trainer through a unified HTTP interface, providing a scalable solution for agent RL training. This design choice reflects a deep understanding of the unique characteristics of rollout and training activities. One of the most innovative features of ProRL Agent is its token-in/token-out trajectory communication, which eliminates the need for re-tokenization and streamlines the training process. Additionally, ProRL Agent offers standardized and extensible sandbox environments that support various agentic tasks in rootless HPC settings. This flexibility allows researchers and developers to train multi-turn LLM agents on complex interactive tasks across different domains such as software engineering, math, STEM, and coding. The infrastructure also supports various reinforcement learning algorithms like PPO and GRPO, making it adaptable to different training scenarios. With its REST API for rollout requests and detailed evaluation metrics like rewards and trajectories, ProRL Agent simplifies the RL training process and enhances long-horizon behavior improvement in multi-turn LLM agents. In conclusion, ProRL Agent is a significant advancement in RL training technology with its "rollout-as-a-service" philosophy that optimizes efficiency, scalability, and maintainability. Its integration into NVIDIA NeMo Gym solidifies its position as a cutting-edge tool for researchers and developers working on complex interactive tasks requiring multi-turn LLM agents.