# SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks

**Authors:** Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, Xian Li  
**arXiv:** [2503.15478](https://arxiv.org/abs/2503.15478)  
**Date:** 2025-03-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2503.15478v1/

## Key points

- Large language model (LLM) agents require effective multi-turn interactions in real-world tasks
- Existing multi-turn reinforcement learning (RL) algorithms struggle with credit assignment over multiple turns and generalization capabilities of LLMs
- Introduction of ColBench benchmark for LLM agent collaboration with human partners in backend programming and frontend design tasks
- Proposal of SWEET-RL algorithm utilizing step-level rewards to enhance policy model performance
- SWEET-RL outperforms other state-of-the-art multi-turn RL algorithms by achieving a 6% absolute improvement in success and win rates on ColBench
- Advancement enables Llama-3.1-8B model to match or surpass GPT4-o performance in collaborative content creation scenarios

## Summary

, , , , 

In the realm of large language model (LLM) agents, the need for effective multi-turn interactions in real-world tasks is paramount. However, existing multi-turn reinforcement learning (RL) algorithms designed to optimize LLM agents often struggle with credit assignment over multiple turns while harnessing the generalization capabilities of LLMs. This challenge has left researchers questioning how to develop algorithms that can effectively address these issues. To delve deeper into this problem, a new benchmark called ColBench has been introduced. In ColBench, an LLM agent collaborates with a human partner across multiple turns to tackle practical tasks in backend programming and frontend design. Building upon this benchmark, a novel RL algorithm known as SWEET-RL (RL with Step-WisE Evaluation from Training-time information) has been proposed. This algorithm utilizes a meticulously crafted optimization objective to train a critic model equipped with additional training-time information. The critic then provides step-level rewards to enhance the policy model. Experimental results have shown that SWEET-RL outperforms other state-of-the-art multi-turn RL algorithms by achieving a 6% absolute improvement in success and win rates on ColBench. This advancement enables the Llama-3.1-8B model to match or even surpass the performance of GPT4-o in collaborative content creation scenarios. The challenges faced by LLM agents in complex sequential decision-making tasks have prompted the exploration of various approaches, including leveraging successful single-turn RLHF algorithms like RAFT and PPO, as well as value function learning methods such as TD-learning. However, these methods often fall short in providing explicit credit assignment across turns or may struggle with sample complexity due to task complexity and long horizons. To address these challenges and pave the way for future research in multi-turn RL algorithms for realistic LLM agent scenarios, the development of ColBench has been instrumental. This benchmark focuses on artifact creation tasks where agents collaborate with humans to produce final artifacts like code or web pages that meet human expectations. By utilizing LLMs as human simulators and implementing functional evaluators for reliable assessments, researchers can now explore more effective strategies for training general, capable, and goal-directed agents within collaborative settings. Overall, the advancements made through SWEET-RL and ColBench represent significant progress towards enhancing the performance of LLM agents in multi-turn interactions across diverse real-world tasks.