# GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation

**Authors:** Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, Hanbo Zhang, Minzhao Zhu  
**arXiv:** [2410.06158](https://arxiv.org/abs/2410.06158)  
**Date:** 2024-10-08  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2410.06158v1/

## Key points

- Introduction of GR-2, a generalist robot agent designed for versatile and generalizable robot manipulation tasks
- Key innovation in GR-2's pre-training process involving exposure to vast number of Internet videos for capturing real-world dynamics
- Large-scale pre-training phase analyzing 38 million video clips and processing over 50 billion tokens to equip GR-2 with generalization abilities
- Fine-tuning for video generation and action prediction using robot trajectories, showcasing impressive multi-task learning capabilities with 97.7% average success rate across 100+ tasks
- Exceptional generalization abilities to new scenarios, including novel backgrounds, environments, objects, and tasks
- Notable performance in bin-picking manipulation with over 100 objects while maintaining robustness with unseen objects
- Correlation between generated video and predicted actions highlighting effectiveness in executing complex manipulation tasks
- Future focus on enhancing generalization capabilities and robustness in action prediction to improve performance in unseen scenarios

## Summary

, , , , 


In this study, we introduce GR-2, a cutting-edge generalist robot agent designed for versatile and generalizable robot manipulation tasks. The key innovation of GR-2 lies in its pre-training process, where it is initially exposed to a vast number of Internet videos to capture the dynamics of the real world. This large-scale pre-training phase involves analyzing 38 million video clips and processing over 50 billion tokens, equipping GR-2 with the ability to generalize across a wide range of robotic tasks and environments during subsequent policy learning. Following the pre-training phase, GR-2 undergoes fine-tuning for both video generation and action prediction using robot trajectories. Through this process, GR-2 demonstrates impressive multi-task learning capabilities, achieving an average success rate of 97.7% across more than 100 different manipulation tasks. Furthermore, GR-2 showcases exceptional generalization abilities to new and previously unseen scenarios, including novel backgrounds, environments, objects, and tasks. One notable highlight of GR-2's performance is its ability to perform bin-picking manipulation with over 100 objects in an end-to-end manner while maintaining remarkable robustness when handling unseen objects. The correlation between the generated video and predicted actions further underscores the effectiveness of GR-2 in understanding and executing complex manipulation tasks. Moving forward, the research team aims to enhance GR-2's generalization capabilities and robustness in action prediction with a specific focus on improving performance in unseen manipulation scenarios. By leveraging state-of-the-art techniques in generative robotic video-language-action modeling, GR-2 represents a significant advancement towards developing a truly versatile and adaptable robot agent for various real-world applications. Overall, the findings presented in this study contribute valuable insights into the field of generalist robot manipulation by showcasing the potential of pre-training models on large-scale datasets to improve generalization and robustness in robotic tasks. The success of GR-2 opens up new possibilities for advancing autonomous robotics technology and paving the way for more sophisticated and capable robotic agents in the future.