This study introduces Tool-RoCo, a groundbreaking benchmark designed to assess the performance of large language models (LLMs) in long-term multi-agent cooperation scenarios. Drawing inspiration from RoCo, a multi-robot cooperative benchmark, Tool-RoCo focuses on agent autonomy by treating other agents as tools and introducing cooperative tools. These tools allow each LLM to select appropriate actions based on the current state, receive feedback, and adjust their selections in subsequent rounds. To evaluate different levels of autonomy, four distinct LLM paradigms are proposed: centralized cooperation, centralized self-organization, decentralized cooperation, and self-organization. Each paradigm offers unique insights into how LLMs interact with one another and coordinate their actions through tool usage. The benchmark includes three challenging multi-robot tasks - SORT, PACK, and CABINET - aimed at measuring format and parameter accuracy as well as agent coordination within the context of tool utilization. The results obtained from various LLMs showcased intriguing patterns. Larger models such as GPT-4.1 and GPT-5 exhibited stronger reflection and modification capabilities compared to smaller models like GPT-4o-mini and GPT-5-mini. Despite achieving higher tool calling accuracy and execution validity rates, the latest GPT-5 model demonstrated lower levels of reflection and modification when compared to GPT-4.1. Furthermore, the impact of different cooperation paradigms was evident in the study's findings. The Centralized Cooperation Paradigm yielded stable tool usage due to centralized coordination by a single LLM agent. On the other hand, decentralized paradigms highlighted challenges faced by smaller models in maintaining parameter validity and execution rates under distributed reasoning settings. Overall, these results underscore the importance of reflective adaptation and complexity within cooperative frameworks for enhancing LLM performance in multi-agent systems. By moving beyond basic tool metrics to explore higher-level cooperative measures such as cooperative tool ratio (CT) and self-organization (SO), Tool-RoCo aims to provide a comprehensive evaluation framework for assessing autonomy and coordination among LLM-based agents in collaborative tasks.
SummaryTool-RoCo is a test to see how well big talking robots work together for a long time. There are four ways the robots can work together: all in one group, some in one group and some on their own, each on their own, or figuring things out on their own. The test includes three hard tasks for the robots to do - sorting things, packing things, and organizing items in a cabinet - to check how well they follow instructions and work together using tools. Big talking robots like GPT-4.1 and GPT-5 can think and change better than smaller ones like GPT-4o-mini and GPT-5-mini. Even though the newest model GPT-5 is good at using tools correctly, it's not as good at changing its thinking compared to older models.
DefinitionsBenchmark: A standard or point of reference used for comparison. Large language models (LLMs): Big talking robots that can understand and generate human-like language. Cooperation paradigms: Different ways that robots can work together towards a common goal. Agent coordination: How well the robots communicate and collaborate with each other. Utilization: Making use of something effectively or efficiently.
Introduction:
Language models have come a long way in recent years, with the development of large language models (LLMs) such as GPT-3 and BERT. These models have shown impressive capabilities in natural language processing tasks, but their performance in multi-agent cooperation scenarios has not been extensively studied. This is where Tool-RoCo comes into play - a new benchmark designed to evaluate the performance of LLMs in long-term cooperative settings.
Inspiration from RoCo:
Tool-RoCo draws inspiration from RoCo, a multi-robot cooperative benchmark that focuses on agent autonomy. In this context, other agents are treated as tools and introduced as cooperative tools to aid each LLM's decision-making process. These tools allow for better coordination among agents by providing feedback and allowing them to adjust their actions accordingly.
Four distinct LLM paradigms:
To evaluate different levels of autonomy, Tool-RoCo proposes four distinct LLM paradigms: centralized cooperation, centralized self-organization, decentralized cooperation, and self-organization. Each paradigm offers unique insights into how LLMs interact with one another and coordinate their actions through tool usage.
Challenging multi-robot tasks:
The benchmark includes three challenging multi-robot tasks - SORT, PACK, and CABINET - aimed at measuring format and parameter accuracy as well as agent coordination within the context of tool utilization. These tasks require agents to work together towards a common goal while utilizing the available tools effectively.
Results obtained from various LLMs:
The results obtained from various LLMs showcased intriguing patterns. Larger models such as GPT-4.1 and GPT-5 exhibited stronger reflection and modification capabilities compared to smaller models like GPT-4o-mini and GPT-5-mini. Despite achieving higher tool calling accuracy and execution validity rates, the latest GPT-5 model demonstrated lower levels of reflection and modification when compared to GPT-4.1. This highlights the importance of model size and complexity in LLM performance.
Impact of different cooperation paradigms:
The study also explored the impact of different cooperation paradigms on LLM performance. The Centralized Cooperation Paradigm yielded stable tool usage due to centralized coordination by a single LLM agent. On the other hand, decentralized paradigms highlighted challenges faced by smaller models in maintaining parameter validity and execution rates under distributed reasoning settings.
Importance of reflective adaptation and complexity:
Overall, these results underscore the importance of reflective adaptation and complexity within cooperative frameworks for enhancing LLM performance in multi-agent systems. By moving beyond basic tool metrics to explore higher-level cooperative measures such as cooperative tool ratio (CT) and self-organization (SO), Tool-RoCo aims to provide a comprehensive evaluation framework for assessing autonomy and coordination among LLM-based agents in collaborative tasks.
Conclusion:
In conclusion, Tool-RoCo is a groundbreaking benchmark that offers valuable insights into the performance of large language models in long-term multi-agent cooperation scenarios. With its focus on agent autonomy, utilization of cooperative tools, and challenging tasks, this benchmark provides a comprehensive evaluation framework for assessing LLMs' capabilities in collaborative settings. The results obtained from various LLMs highlight the importance of model size, complexity, and reflective adaptation for achieving optimal performance. Future research can build upon this benchmark to further improve our understanding of how language models can effectively collaborate with one another towards common goals.
This study proposes Tool-RoCo, a novel benchmark for evaluating large language models (LLMs) in long-term multi-agent cooperation based on RoCo, a multi-robot cooperative benchmark. Recent research on LLM-based multi-agent systems has relied on predefined orchestration, while ignoring agent autonomy. Tool-RoCo treats other agents as tools and introduces cooperative tools, leveraging tool usage to evaluate multi-agent cooperation and self-organization. Tool usage means that each agent (LLM) selects a tool from a candidate set based on the current state, receives feedback, and adjusts its selection in subsequent rounds. To evaluate different autonomy levels, we propose four LLM paradigms: (1) centralized cooperation, where a single LLM allocates tools to all agents; (2) centralized self-organization, where a central LLM autonomously activates agents while keeping others inactive; (3) decentralized cooperation, where each agent has its own LLM and calls tools based on local information; and (4) self-organization, where a randomly chosen initial agent can request collaboration, activating additional agents via tool calls. Tool-RoCo includes three multi-robot tasks, SORT, PACK, and CABINET, to measure format and parameter accuracy and agent coordination through tool usage. The results using several LLMs showed that cooperative tools accounted for only 7.09% of all tools, indicating that LLM-based agents rarely invoked others as assistants. Moreover, activation tools accounted for 96.42%, suggesting that current LLMs tend to maintain active agents while seldom deactivating them for adaptive coordination. Tool-RoCo provides a systematic benchmark to evaluate LLM autonomy and cooperation in multi-agent tasks. Code and Demo: https://github.com/ColaZhang22/Tool-Roco
