# Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning

**Authors:**  NVIDIA,  :, Alisson Azzolini, Hannah Brandon, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, Francesco Ferroni, Rama Govindaraju, Jinwei Gu, Siddharth Gururani, Imad El Hanafi, Zekun Hao, Jacob Huffman, Jingyi Jin, Brendan Johnson, Rizwan Khan  
**arXiv:** [2503.15558](https://arxiv.org/abs/2503.15558)  
**Date:** 2025-03-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2503.15558v1/

## Key points

- Models aim to understand the physical world and make decisions through long chain-of-thought reasoning processes
- Focus on key capabilities for physical common sense and embodied reasoning
- Utilization of hierarchical ontology for representing fundamental knowledge about space, time, and physics
- Two-dimensional ontology used for embodied reasoning across different physical embodiments
- Development of Cosmos-Reason1 models involves data curation and training in four stages: vision pre-training, general supervised fine-tuning (SFT), Physical AI SFT, and Physical AI reinforcement learning (RL)
- Evaluation benchmarks built according to ontologies show significant improvements with Physical AI SFT and reinforcement learning
- Rule-based cleaning and rewriting stage implemented to produce valid SFT samples for reasoning annotations
- Curated datasets include free-form questions from high-quality video clips with human-annotated captions, as well as multiple-choice questions to test model capabilities
- Detailed descriptions used to construct understanding MCQs and reasoning MCQs
- Curation pipeline applied across various datasets such as BridgeData V2 for robotic manipulation behaviors and RoboVQA for robotics-focused visual question answering
- Each dataset presents unique challenges related to physical common sense and embodied reasoning

## Summary

The  models aim to understand the physical world and generate appropriate embodied decisions through long chain-of-thought reasoning processes. The focus is on key capabilities for , including  and . To represent physical common sense, a hierarchical ontology capturing fundamental knowledge about space, time, and physics is utilized. For embodied reasoning, a two-dimensional ontology generalizing across different physical embodiments is employed. The development of two multimodal large language models, Cosmos-Reason1-8B and Cosmos-Reason1-56B, involves data curation and training in four stages: vision pre-training, general supervised fine-tuning (SFT), Physical AI SFT, and Physical AI reinforcement learning (RL) as post-training. Evaluation benchmarks for physical common sense and embodied reasoning are built according to the ontologies, showing significant improvements with Physical AI SFT and reinforcement learning. Furthermore, a rule-based cleaning and rewriting stage is implemented to produce valid SFT samples for reasoning annotations. The curated datasets include free-form questions from high-quality video clips with human-annotated captions, as well as multiple-choice questions to test model capabilities. Additionally, detailed descriptions are used to construct understanding MCQs and reasoning MCQs. The curation pipeline is applied across various datasets such as BridgeData V2 for robotic manipulation behaviors and RoboVQA for robotics-focused visual question answering. Each dataset presents unique challenges and tasks related to physical common sense and embodied reasoning. Overall,the refined summary highlights the comprehensive approach taken in developing the Cosmos-Reason1 models for advancing Physical AI research.