Nvidia comes out cosmos-reason1: a series of models of AI advances the physical sense and the reasoning embodied in real environments

The AI ​​has progressed in the treatment of languages, mathematics and the generation of code, but the extension of these capacities to physical environments remains difficult. Physical AI seeks to fill this gap by developing systems that perceive, include and act in dynamic and real parameters. Unlike conventional AI which deals with text or symbols, physical AI engages with sensory entries, in particular video, and generates responses based on real world physics. These systems are designed for navigation, manipulation and interaction, based on common sense reasoning and an embodied understanding of space, time and physical laws. The applications cover robotics, autonomous vehicles and human-machine collaboration, where adaptability to real-time perception is crucial.

The current low connection of AI models to real world physics is a major limitation. Although they succeed in the abstract tasks, they often do not manage to predict the physical consequences or to respond appropriately to sensory data. Concepts such as severity or spatial relationships are not intuitively understood, which makes them unreliable for embodied tasks. Training directly in the physical world is expensive and risky, which hinders development and iteration. This lack of physical and embodied understanding is an important obstacle to the effective deployment of AI in real world applications.

Previously, the tools of physical reasoning in AI were fragmented. Vision-Langus models have linked visual and textual data but lacked the depth of reasoning. The rules based on rules were rigid and failed in new scenarios. Simulations and synthetic data often lack the nuances of real world physics. Above all, there was no standardized framework to define or assess the common sense physical or embodied reasoning. The inconsistent methodologies and benchmarks have made progress difficult to quantify. Reinforcement learning approaches lacked specific task reward structures, leading to models that fought with cause and effective reasoning and physical feasibility.

Nvidia researchers have introduced Cosmos-reason1A series of large -language multimodal models. These models, Cosmos-reason1-7b And Cosmos-reason1-56bwere designed specifically for physical reasoning tasks. Each model is formed in two major phases: a final setting (SFT) and a physical strengthening learning (RL). What differentiates this approach is the introduction of a double -allowed system. A hierarchical ontology organizes the physical common sense in three main categories, space, time and fundamental physics, divided into 16 subcategories. The second ontology is two -dimensional and cartoing of reasoning capacities through five embodied agents, including humans, robot weapons, humanoid robots and autonomous vehicles. These ontologies are training guides and evaluation tools to compare the physical reasoning of AI.

The Cosmos-Reason1 architecture uses an LLM only a decoder increased by a vision encoder. The videos are processed to extract visual features, which are then projected in a space shared with language tokens. This integration allows the model to reason on textual and visual data simultaneously. The researchers organized a set of massive data comprising around 4 million pairs of video text annotated for training. These include action descriptions, multiple questions and traces of long chain reasoning. The strengthening learning phase is motivated by verifiable rewards based on rules derived from multiple choice questions marked by humans and self-supervised video tasks. These tasks include the prediction of the temporal direction of videos and the resolution of puzzles with space-time patches, which makes training deeply linked to the physical logic of the real world.

The team built three benchmarks for physical sense, space, time and fundamental physics, containing 604 questions of 426 videos. Six benchmarks were built for an embodied reasoning with 610 questions from 600 videos, covering a wide range of tasks. The Cosmos-Reason1 models have outperformed the previous basic lines, in particular after the RL phase. They have notably improved in the verification of the completion of tasks, predicting the next plausible actions and evaluating the physical feasibility of actions. These gains were observed in the two model sizes, the cosmos-reash1-56b showing stronger performance in most metrics. This performance improvement highlights the effectiveness of the use of structured ontologies and multimodal data to improve physical reasoning in AI.

Several key points to remember from research on Cosmos-Reash1:

• Two models introduced: Cosmos-Reash1-7b and Cosmos-Reason1-56b, formed specifically for physical reasoning tasks.
• The models have been formed in two phases: a fine adjustment to refinement (SFT) and learning in physical strengthening (RL).
• The training data set includes around 4 million pairs of annotated video texts organized for physical reasoning.
• Reinforcement learning uses rewards based on rules and verifiable, derived from human annotations and video tasks.
• The team relied on two ontologies: a hierarchical with three categories and 16 subcategories, and two-dimensional cartography agent capacities.
• Benchmarks: 604 questions of 426 videos for common sense physical and 610 of 600 videos for embodied reasoning.
• Performance gains were observed in all references after the RL training, in particular to predict the following actions and the verification of the completion of tasks.
• Applicability of the real world for robots, vehicles and other embodied agents in various environments.

In conclusion, the Cosmos-Reason1 initiative shows how AI can be better equipped for the physical world. It deals with the key limits of perception, reasoning and decision -making that have hampered progress in the deployment of AI in incarnated scenarios. The structured training pipeline, based on real world data and ontological frames, guarantees that the models are precise and adaptable. These advances indicate a major advance in filling the gap between the abstract reasoning of AI and the needs of systems which must operate in unpredictable and real environments.

---

Discover the Paper,, Project page,, Models on the embraced face And GitHub page. All the merit of this research goes to researchers in this project. Also, don't hesitate to follow us Twitter And don't forget to join our 95K + ML Subdreddit and subscribe to Our newsletter.