# Open X-Embodiment: Robotic Learning Datasets and RT-X Models

**Authors:** Dorsa Sadigh, Danny Driess, Fei Xia, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Yevgen Chebotar, Pierre Sermanet, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Igor Mordatch, Dinesh Jayaraman, Wenxuan Zhou, Nicolas Heess, Tianli Ding, Trevor Darrell, Chelsea Finn  
**arXiv:** [2310.08864](https://arxiv.org/abs/2310.08864)  
**Date:** 2023-10-13  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2310.08864v8/

## Key points

- Shift towards using large, high-capacity models trained on diverse datasets in robotics
- Development of pretrained models serving as general backbones for different robotic tasks
- Growing interest in developing generalist X-robot policies adaptable to new robots, tasks, and environments
- Recent study assembling a dataset from 22 different robots demonstrating 527 skills across 160266 tasks
- Training of high-capacity model RT-X on this data showing positive transferability among multiple robots
- Consideration of two model architectures in experiments: RT-1 (Transformer-based) and RT-2 (vision-language model)
- Both models take visual input and natural language instructions to output tokenized actions
- Exploration of how X-embodiment training can enhance the performance of learned policies on individual robots
- Research aims to explore benefits of consolidating pretrained models in robotics for efficient adaptation across various platforms
- Experimental results showcasing effective X-robot policies with positive transferability and improved performance across multiple robotic platforms

## Summary

The field of robotics has seen a shift towards using large, high-capacity models trained on diverse datasets to efficiently tackle various downstream applications. This approach has led to the development of pretrained models that serve as general backbones for different robotic tasks. The question now arises: can we consolidate these pretrained models in robotics as well? Traditionally, robotic learning methods involve training separate models for each application, robot, and environment. However, there is a growing interest in developing generalist X-robot policies that can be adapted easily to new robots, tasks, and environments. To explore this possibility in the context of robotic manipulation, datasets in standardized formats and models have been provided in a recent study. The study involved assembling a dataset from 22 different robots collected through a collaboration between 21 institutions. This dataset demonstrated 527 skills across 160266 tasks. A high-capacity model called RT-X was trained on this data and showed positive transferability by improving the capabilities of multiple robots through leveraging experience from other platforms. Two model architectures were considered in the experiments: RT-1 and RT-2. RT-1 is an efficient Transformer-based architecture designed for robotic control, while RT-2 is a large vision-language model co-fine-tuned to output robot actions as natural language tokens. Both models take visual input and natural language instructions describing tasks and output tokenized actions. RT-X was designed to evaluate how X-embodiment training can enhance the performance of learned policies on individual robots. The study built upon the Transformer-based robotic policies of RT-1 and RT-2, adapting them to the X-embodiment setting for experimentation. Overall, this research aims to explore the potential benefits of consolidating pretrained models in robotics by providing standardized datasets and models for efficient adaptation across different robots, tasks, and environments. The experimental results showcased effective X-robot policies that demonstrate positive transferability and improved performance across multiple robotic platforms. More information about this project can be found on their website at https://robotics-transformer-x.github.io.