# RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

**Authors:** Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan  
**arXiv:** [2307.15818](https://arxiv.org/abs/2307.15818)  
**Date:** 2023-07-28  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2307.15818v1/

## Key points

- Authors explore integration of vision-language models trained on web data into robotic control systems
- Objective is to enhance adaptability and cognitive capabilities in robotic systems
- Approach involves co-fine-tuning vision-language models using robotic trajectory data and visual question answering
- Robotic actions represented as text tokens alongside natural language tokens in training set
- Novel category of models termed RT-2, showcasing high-performing robotic policies and diverse emergent capabilities
- Advancements include enhanced generalization, interpretation of novel commands, rudimentary reasoning abilities, and multi-stage semantic reasoning tasks
- Incorporation of chain of thought reasoning mechanisms enhances model's proficiency in various tasks

## Summary

In their paper titled "RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control," authors Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence,
Chuyuan Fu,Montse Gonzalez Arenas,Keeerthana Gopalakrishnan,Kehang Han,Karol Hausman,Alexander Herzog and others explore the integration of vision-language models trained on  into end-to-end robotic control systems. The primary objective is to enhance  and facilitate  within robotic systems by leveraging large-scale pretraining on language and vision-language datasets sourced from the web. The proposed approach involves co-fine-tuning cutting-edge vision-language models using both robotic trajectory data and , like visual question answering. Unlike conventional methods in the field,the authors advocate for a straightforward yet effective strategy wherein robotic actions are represented as text tokens and seamlessly integrated into the model's training set alongside natural language tokens. This novel category of models is termed , with RT-2 serving as a concrete instantiation of this concept. Extensive evaluation comprising 6k trials demonstrates that this methodology yields high-performing robotic policies while empowering RT-2 with a diverse array of emergent capabilities derived from . These advancements include significantly enhanced generalization towards novel objects, adept interpretation of commands absent in the robot's training data (e.g., placing an object on a specific number or icon), and rudimentary reasoning abilities in response to user directives (such as identifying the smallest or largest object in proximity). Moreover, by incorporating chain of thought reasoning mechanisms into RT-2's architecture, the model showcases proficiency in multi-stage semantic reasoning tasks. For instance, it can deduce which object to pick up for use as an improvised hammer (e.g., a rock) or determine the most suitable type of drink for an individual feeling fatigued (an energy drink). Overall, this research underscores how integrating  into robotic control systems can yield substantial improvements in adaptability and cognitive capabilities.