# PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks

**Authors:** Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner, Eric Undersander, Tsung-Yen Yang  
**arXiv:** [2411.00081](https://arxiv.org/abs/2411.00081)  
**Date:** 2024-10-31  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2411.00081v1/

## Key points

- The PARTNR benchmark is designed to study human-robot collaboration in household activities.
- It assesses coordination between humans and robots in tasks with spatial, temporal, and heterogeneous agent capability constraints.
- Tasks are generated using Large Language Models (LLMs) and verified through simulation.
- PARTNR consists of 100,000 natural language tasks across 60 houses with 5,819 unique objects, making it the largest benchmark of its kind.
- Analysis of state-of-the-art LLMs on PARTNR tasks reveals limitations such as poor agent coordination, task tracking difficulties, and error recovery challenges.
- When LLMs collaborate with real humans, they require more steps compared to human-only or two-human collaborations.
- Fine-tuning smaller LLMs with planning data can lead to performance levels comparable to much larger models while also being significantly faster during inference processes.
- The PARTNR benchmark aims to identify and address challenges faced by collaborative embodied agents, driving research towards enhancing the capabilities of collaborative robots in real-world settings.

## Summary

The PARTNR benchmark has been introduced to study human-robot collaboration in household activities. It assesses the coordination between humans and robots in everyday tasks that involve spatial, temporal, and heterogeneous agent capability constraints. The tasks are generated through a semi-automated pipeline using Large Language Models (LLMs) and are verified through simulation. With 100,000 natural language tasks spanning across 60 houses and involving 5,819 unique objects, PARTNR is the largest benchmark of its kind. A comprehensive analysis of state-of-the-art LLMs on PARTNR tasks has revealed significant limitations in current models. These include poor coordination between agents, difficulties in task tracking, and challenges in recovering from errors. When LLMs are paired with real humans for collaborative tasks, they require 1.5 times as many steps as two humans working together and 1.1 times more steps than a single human would need. This highlights the potential for improvement in these models to enhance their efficiency and effectiveness. The study also demonstrates that fine-tuning smaller LLMs with planning data can lead to performance levels comparable to models that are nine times larger while also being significantly faster during inference processes by a factor of 8.6 times. In conclusion, the PARTNR benchmark serves as a critical tool for identifying and addressing the substantial challenges faced by collaborative embodied agents. By shedding light on these challenges and showcasing areas where improvements can be made within existing models, this benchmark aims to drive further research towards enhancing the capabilities of collaborative robots in real-world settings.