Introducing NVIDIA Cosmos 3: The Open Model That Thinks, Generates, and Acts
Physical AI faces an unbounded, unpredictable real world and real-world data cannot scale; the talk reframes this by treating compute as data. NVIDIA Cosmos is presented as an open omnimodel for physical AI built on a new mixture-of-transformers architecture: an autoregressive transformer that reasons, plans, and instructs, and a diffusion transformer that generates what comes next from pixels, action, sound, and language. Post-training across embodiments converts Cosmos into an action-conditioned world model that can describe scenes, generate physics-accurate synthetic video, close the loop for policy training and evaluation, and produce actions for robots; the approach trades data collection for compute-generated synthetic data and computer-generated teachers.
Key technical points
- Problem statement: the real world is infinite and unpredictable, and real-world data is impossible to scale for physical AI.
- Principle: 'compute is data'—use compute to generate the data and supervision needed for physical AI instead of relying solely on collected real-world data.
- Model class: Cosmos is an open 'omnimodel' for physical AI built on a new mixture-of-transformers architecture.
- Architecture decomposition: a pipeline of transformers including an autoregressive transformer and a diffusion transformer.
- Inputs to the system: pixels, action, sound, and language are fed into the autoregressive transformer.
- Role of the autoregressive transformer: reasons, plans, and issues instructions to the diffusion transformer (serves as the planner/instructor).
- Role of the diffusion transformer: generates the next content (e.g., future frames or observations) conditioned on instructions from the autoregressive transformer.
- Post-training workflow: developers post-train Cosmos across different embodiments and use cases to adapt it to specific robots or environments.
- Vision-language capability: as a VLM, Cosmos watches the physical world, understands and describes scenes, and flags what matters.
- World-model capability: Cosmos can generate physics-accurate synthetic video from an image, text, or video input.
- Simulator capability and training loop: Cosmos can act as a simulator that closes the loop for policy training and evaluation (enables closed-loop, action-conditioned training).
- Action-conditioned prediction: as the foundation of NVIDIA Omnidreams, Cosmos predicts future frames iteratively (frame by frame) conditioned on actions.
- Deployment outcome: after post-training, Cosmos becomes a world-action model that perceives, reasons, plans, and generates actions for 'robots of every kind'—for everything that moves.
- New data/teacher paradigm: Cosmos enables a new kind of data and a new kind of teacher generated by computer (synthetic data and computer-generated supervision).
For engineering teams, the talk signals a shift from costly real-world data collection toward heavy use of compute-generated synthetic data and simulation. Architecturally, plan for a two-stage transformer pipeline: an autoregressive planner/instructor and a diffusion generator, with multimodal inputs (pixels, action, sound, language) and multimodal outputs (descriptions, videos, actions). Expect development flows that require post-training across embodiments to adapt policies and perception to target robots, and closed-loop simulation to train and evaluate policies. The presentation does not specify evaluation metrics, safety guardrails, or detailed deployment mechanics, so teams must design their own validation, metrics, and safeguards around physics fidelity, sim-to-real transfer, and runtime action safety. Operationally, this approach implies substantial compute resources and orchestration for synthetic-data generation, post-training, and iterative policy evaluation.
Cosmos's emphasis on simulation-driven, action-conditioned world models is directly relevant to Japan's robotics and automation sectors: manufacturing automation, mobile robotics, and service robots can leverage post-trained omnimodels that generate physics-accurate synthetic video and train policies in closed-loop simulators. Japanese engineering teams should prepare pipelines for heavy compute-based data generation, embodiment-specific post-training, and rigorous sim-to-real validation, while acknowledging the talk does not provide concrete metrics or safety/production guardrails—these must be defined locally to meet industry and regulatory requirements in Japanese deployments.