Results of the summarizing process for the arXiv paper: 2503.15558v1

The models aim to understand the physical world and generate appropriate embodied decisions through long chain-of-thought reasoning processes. The focus is on key capabilities for , including and . To represent physical common sense, a hierarchical ontology capturing fundamental knowledge about space, time, and physics is utilized. For embodied reasoning, a two-dimensional ontology generalizing across different physical embodiments is employed. The development of two multimodal large language models, Cosmos-Reason1-8B and Cosmos-Reason1-56B, involves data curation and training in four stages: vision pre-training, general supervised fine-tuning (SFT), Physical AI SFT, and Physical AI reinforcement learning (RL) as post-training. Evaluation benchmarks for physical common sense and embodied reasoning are built according to the ontologies, showing significant improvements with Physical AI SFT and reinforcement learning. Furthermore, a rule-based cleaning and rewriting stage is implemented to produce valid SFT samples for reasoning annotations. The curated datasets include free-form questions from high-quality video clips with human-annotated captions, as well as multiple-choice questions to test model capabilities. Additionally, detailed descriptions are used to construct understanding MCQs and reasoning MCQs. The curation pipeline is applied across various datasets such as BridgeData V2 for robotic manipulation behaviors and RoboVQA for robotics-focused visual question answering. Each dataset presents unique challenges and tasks related to physical common sense and embodied reasoning. Overall,the refined summary highlights the comprehensive approach taken in developing the Cosmos-Reason1 models for advancing Physical AI research.

• - Models aim to understand the physical world and make decisions through long chain-of-thought reasoning processes
• - Focus on key capabilities for physical common sense and embodied reasoning
• - Utilization of hierarchical ontology for representing fundamental knowledge about space, time, and physics
• - Two-dimensional ontology used for embodied reasoning across different physical embodiments
• - Development of Cosmos-Reason1 models involves data curation and training in four stages: vision pre-training, general supervised fine-tuning (SFT), Physical AI SFT, and Physical AI reinforcement learning (RL)
• - Evaluation benchmarks built according to ontologies show significant improvements with Physical AI SFT and reinforcement learning
• - Rule-based cleaning and rewriting stage implemented to produce valid SFT samples for reasoning annotations
• - Curated datasets include free-form questions from high-quality video clips with human-annotated captions, as well as multiple-choice questions to test model capabilities
• - Detailed descriptions used to construct understanding MCQs and reasoning MCQs
• - Curation pipeline applied across various datasets such as BridgeData V2 for robotic manipulation behaviors and RoboVQA for robotics-focused visual question answering
• - Each dataset presents unique challenges related to physical common sense and embodied reasoning

SummaryModels are like tools that help us understand and make decisions about the world around us. They use a special way of thinking to figure things out. Models focus on important skills for understanding how things work in the real world. They organize information about space, time, and physics in a structured way. To teach models to be smarter, we train them using different stages and tests. Definitions- Models: Tools or systems designed to understand and make decisions about the physical world. - Reasoning: Thinking carefully and logically to come up with answers or solutions. - Ontology: A way of organizing knowledge or information into categories. - Embodied reasoning: Understanding how things work based on physical experiences or interactions. - Dataset: A collection of data or information used for training models or conducting experiments.

The field of Artificial Intelligence (AI) has made significant strides in recent years, with advancements in natural language processing and computer vision allowing machines to understand and interact with the world around them. However, one area that still presents challenges for AI is embodied reasoning - the ability to make decisions based on physical common sense and understanding of the environment. In order to address this gap, a team of researchers from OpenAI have developed two large language models: Cosmos-Reason1-8B and Cosmos-Reason1-56B. These models aim to understand the physical world and generate appropriate embodied decisions through long chain-of-thought reasoning processes. The focus is on key capabilities for Physical AI, including physical common sense and embodied reasoning. To represent these concepts, the researchers utilized a hierarchical ontology capturing fundamental knowledge about space, time, and physics. This allows the models to have a foundational understanding of how objects behave in different environments. In addition to this ontology for physical common sense, a two-dimensional ontology was also employed for embodied reasoning. This generalizes across different physical embodiments such as humans or robots, allowing the models to reason about actions and behaviors regardless of their form. The development of these multimodal large language models involved four stages: vision pre-training, general supervised fine-tuning (SFT), Physical AI SFT, and Physical AI reinforcement learning (RL) as post-training. The first stage involves training on large-scale visual datasets such as ImageNet or COCO in order to develop basic visual understanding capabilities. The second stage focuses on supervised fine-tuning using text-based tasks such as question answering or summarization. However, it is at the third stage where things get interesting - Physical AI SFT takes into account both visual data from images/videos as well as textual information from captions or descriptions. This allows the model to learn not only from text but also from real-world visual input which is crucial for developing an understanding of embodiment and physical common sense. The final stage, Physical AI RL, involves reinforcement learning which allows the model to make decisions based on rewards or punishments. This post-training step further improves the models' abilities for embodied reasoning by allowing them to learn from their own experiences and interactions with the environment. To evaluate the performance of these models, benchmarks were built according to the ontologies for physical common sense and embodied reasoning. The results showed significant improvements with Physical AI SFT and reinforcement learning compared to general supervised fine-tuning alone. In order to train these large language models, a curated dataset was necessary. This involved data curation from various sources such as high-quality video clips with human-annotated captions, as well as multiple-choice questions (MCQs) to test model capabilities. To ensure valid samples for reasoning annotations, a rule-based cleaning and rewriting stage was implemented. The curated datasets include free-form questions from BridgeData V2 for robotic manipulation behaviors and RoboVQA for robotics-focused visual question answering. Each dataset presents unique challenges and tasks related to physical common sense and embodied reasoning, highlighting the need for comprehensive training in this area. Overall, the development of Cosmos-Reason1 models represents a significant step forward in advancing Physical AI research. By incorporating both visual data and textual information into their training process, these models have shown promising results in understanding physical common sense and making appropriate decisions based on embodiment. With continued advancements in this field, we can expect even more sophisticated AI systems that are capable of navigating complex real-world environments with ease.