World models as simulators, data engines and RL environments

Why this chapter exists

The families so far all produce a policy: something that, given what the robot sees, says what to do. This chapter is about models that do not. A world model as a simulator answers a different question: if this were done, what would happen next? Other models then use the answer. A planner tries candidate actions in it. A data pipeline asks it to imagine thousands of demonstrations nobody recorded. A reinforcement learning (RL) algorithm trains a policy inside it, as if it were a video game. An evaluator runs a policy in it before letting the policy near a real robot.

The motivation is economic and practical. Real robot interaction is slow, expensive and, for RL, risky: a policy that is still learning breaks things. Physics simulators are fast and safe but narrow, and their images and contact physics never quite match reality. Learned world models promise the breadth of real video with the safety of a simulator, at a fidelity cost that is the subject of most of this chapter.

After this chapter you will be able to:

• name the four roles a world model plays for robot learning, and place a new paper in one of them;
• explain with figure 8.3 how a policy trained in imagination exploits the model's mistakes, and two ways to limit it;
• tell a plausible rollout from a feasible one with figure 8.4, and say why image-quality metrics do not tell them apart.

The mechanism

A world model in this chapter is a forward dynamics model in the sense of chapter 4. It predicts the next observation from what has been seen and what is done:

Two features make the modern versions different from a physics engine. The observation $o$ is camera video, so the model can be trained on footage rather than on hand-built scenes. And $W$ is a large generative model, usually a video diffusion model, pretrained on internet-scale video and then adapted to robots. NVIDIA's Cosmos-Predict2.5 is the clearest open example: trained on 200 million curated video clips, released at 2 and 14 billion parameters, and built to support "synthetic data generation, policy evaluation, and closed-loop simulation" (NVIDIA et al. 2025, arXiv 2511.00062). Genie showed that the actions themselves can be latent: trained on unlabelled internet video, it lets a user act frame by frame in worlds it generates (Bruce et al. 2024, arXiv 2402.15391).

The idea of learning a world model and acting inside it is older than any of these. PlaNet planned in a learned latent model in 2018, Dreamer learned behaviours by imagining in it, and DreamerV3 outperformed specialised methods across more than 150 tasks with a single configuration, including collecting diamonds in Minecraft from scratch (Hafner et al. 2018, arXiv 1811.04551; Hafner et al. 2019, arXiv 1912.01603; Hafner et al. 2023, arXiv 2301.04104). What changed in 2025 and 2026 is the observation: real camera video of real robots, at a quality good enough that a policy trained on it can transfer.

Role 1: the data engine

The most widely used role is generating training data. Video models generate pixels, not actions, so every synthetic-data pipeline has the same shape: generate a video, then recover the actions that would have produced it with an inverse dynamics model,

and train the policy on the result as if it were a demonstration. DreamGen made the recipe explicit in four stages (Jang et al. 2025, arXiv 2505.12705), drawn in figure 8.2.

The results are what made the role popular. DreamGen reports that a humanoid learned 22 new behaviours, in seen and unseen environments, from teleoperation data of a single pick-and-place task in one environment (Jang et al. 2025, arXiv 2505.12705). NVIDIA's synthetic motion blueprint generated 780,000 trajectories, which it equates to 6,500 hours of human demonstration, in 11 hours, and reports improving GR00T N1's performance by 40 percent when combining them with real datavendor claim (NVIDIA Newsroom 2025). GigaWorld-0 is designed explicitly as a data engine, combining video generation with 3D reconstruction and physical system identification. Its makers report that GigaBrain-0, trained on its data, performs strongly on physical robots "without any real-world interaction during training" (GigaWorld Team et al. 2025, arXiv 2511.19861; GigaBrain Team et al. 2025, arXiv 2510.19430). Cosmos-Transfer does the related job of style transfer: it turns simulator renderings or segmentation maps into realistic video, for sim-to-real and real-to-real augmentation (NVIDIA et al. 2025, arXiv 2503.14492).

Role 2: the RL environment

Imitation learning copies what demonstrators did. To improve beyond them, a policy has to try things and learn from the outcome, which is RL. On a real robot that means thousands of attempts, resets and risk. Inside a world model it means compute. The policy acts, the model predicts what happens, a reward model scores it, and the policy updates. The catch fits in one line. The policy maximises the return the model predicts, $\hat J(\pi)$, not the real return $J(\pi)$, and the difference is the model's bias:

A policy that is good at maximising $\hat J$ is, among other things, good at finding where $\hat J$ is too high. Figure 8.3 shows how quickly that happens.

With logged pushes only up to a strength of 0.4, the world model is a straight line through a curve that bends upwards, and it has never seen the block fall. Inside the model, the best push is about 0.85: the predicted slide lands on the mark and the imagined reward is 0.99. In the real world a push of 0.85 sends the block off the table. Nothing in the training signal warned the policy, because the training signal was the model.

The two fixes in the figure are the two families of fixes in the literature. Stay near the data: limit the policy to where the model has evidence, which is safe but stops short of the best push. Fix the data: collect where the policy now goes, including failures, and retrain the model. With logged pushes up to 1.0 the model learns where the edge is, and the policy lands close to the mark. World-VLA-Loop builds that second fix into the algorithm. It curates successful and "near-success" trajectories so the model learns subtle failures, predicts rewards from the same latents as the video, and feeds each improved policy's rollouts back into the world model (Liu et al. 2026, arXiv 2602.06508). VLA-MBPO attacks multi-view consistency and error compounding directly (Zhang et al. 2026, arXiv 2603.20607). A May 2026 study argues the deeper problem is a misalignment between search and value learning, and unifies them through diffusion policies (Cheng et al. 2026, arXiv 2605.26282).

The results so far are encouraging. World4RL refines pretrained policies entirely inside a frozen diffusion world model (Jiang et al. 2025, arXiv 2509.19080). WMPO runs on-policy RL for VLAs without touching the real environment and reports emergent self-correction (Zhu et al. 2025, arXiv 2511.09515). DreamPlan fine-tunes a vision-language planner inside a video world model trained on the planner's own suboptimal exploration (Jia et al. 2026, arXiv 2603.16860). WAM-OPD, from August 2026, swaps sparse-reward RL for dense supervision from a teacher on the student's own rollouts. Its authors present early RoboTwin results as "an initial capability proof rather than evidence of broad or uniform generalization" (Yang et al. 2026, arXiv 2608.22364).

Role 3: the evaluator

Evaluating a policy on a real robot is the slowest step in robot learning, and chapter 15 is about doing it well. A world model that predicted real success rates would make evaluation cheap. The Interactive World Simulator runs stable interactions for more than 10 minutes at 15 frames per second on a single RTX 4090. Its authors report that policies trained on its generated data perform comparably to those trained on the same amount of real data, and that success in the model correlates strongly with success in the real world (Wang et al. 2026, arXiv 2603.08546). DreamDojo lists policy evaluation among its applications (Gao et al. 2026, arXiv 2602.06949).

The warning comes from WorldArena, which evaluated 14 world models as data engines, policy evaluators and planners and found "a significant perception-functionality gap": high visual quality "does not necessarily translate into strong embodied task capability" (Shang et al. 2026, arXiv 2602.08971). Figure 8.4 shows the simplest version of that gap.

Role 4: the planner

Chapter 6 covered planning with a world model in detail: imagine candidate futures, score them, act on the best. The same risk applies as in RL, since the planner chooses whichever candidate the model scores highest. V-JEPA 2-AC, from chapter 7, plans in feature space towards image goals (Assran et al. 2025, arXiv 2506.09985), and Cosmos Policy can run as a policy or as a planner (Kim et al. 2026, arXiv 2601.16163).

Plausible is not feasible

A video model is trained to produce frames that look like real video. Nothing in that objective says the motion between frames must be achievable by a particular robot. EVA calls the mismatch "the executability gap": visually coherent rollouts that "violate rigid-body and kinematic consistency" and decode into unstable or infeasible commands (Wang et al. 2026, arXiv 2603.17808). Its fix is to turn the inverse dynamics model into a reward. Generated videos are scored by the smoothness and legality of the actions they imply, which stays informative "even when generated videos contain severe visual artifacts" (Wang et al. 2026, arXiv 2603.17808). Exercise 8.3 implements the simplest version of that check.

Good at, bad at

What it is good at. It multiplies scarce robot data, enables RL without physical risk, and supports evaluation before deployment. It is also the one family whose output a person can watch: a generated rollout can be inspected frame by frame, which a latent vector (chapter 7) cannot.

What it is bad at. Model bias propagates into whatever is trained in the model. Plausible is not feasible. Multi-view consistency remains weak, which is why VLA-MBPO needs a dedicated mechanism for it (Zhang et al. 2026, arXiv 2603.20607). And it is compute-heavy: generating video is the most expensive operation in this book, which matters when RL needs millions of steps.

In code

The crate's world is one dimension wide, which is what makes the failure easy to see. The true task, the model and the logged data fit in a few lines.

/// A robot pushes a block towards a target mark near the table's edge. `theta` in [0, 1] is how
/// hard it pushes. Past the edge the block falls off.
pub const TARGET: f64 = 0.6;
pub const EDGE: f64 = 0.8;

/// How far the block really slides: faster than proportional, because friction matters less at speed.
pub fn true_distance(theta: f64) -> f64 {
    0.3 * theta + 0.9 * theta * theta
}

/// Reward for a slide of `d`: 1 on the mark, falling off linearly, and -1 if the block falls.
pub fn reward(d: f64) -> f64 {
    if d > EDGE { -1.0 } else { 1.0 - (d - TARGET).abs() / TARGET }
}

pub fn true_reward(theta: f64) -> f64 {
    reward(true_distance(theta))
}

/// A learned world model: a straight-line fit of distance against push strength, plus the edge
/// if the data ever showed the block falling.
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct WorldModel {
    pub intercept: f64,
    pub slope: f64,
    /// Push strength beyond which the model predicts a fall; `None` if it never saw one.
    pub edge_theta: Option<f64>,
}

impl WorldModel {
    pub fn distance(&self, theta: f64) -> f64 {
        self.intercept + self.slope * theta
    }
    /// The reward the policy sees when it trains inside the model.
    pub fn reward(&self, theta: f64) -> f64 {
        match self.edge_theta {
            Some(e) if theta > e => -1.0,
            _ => reward(self.distance(theta).min(EDGE)),
        }
    }
}

/// Logged pushes the world model is trained on: `n` evenly spaced strengths from `lo` to `hi`.
pub fn collect(lo: f64, hi: f64, n: usize) -> Vec<(f64, f64)> {
    (0..n)
        .map(|i| {
            let th = if n == 1 { lo } else { lo + (hi - lo) * i as f64 / (n - 1) as f64 };
            (th, true_distance(th))
        })
        .collect()
}

Exercises

The stubs are in rust/ch08-simulators/src/exercises.rs.

cargo test -p ch08-simulators --test exercises ex8_1

cargo test -p ch08-simulators --test exercises ex8_2

cargo test -p ch08-simulators --test exercises ex8_3

Further reading

• DreamGen, the clearest description of the synthetic-data recipe (Jang et al. 2025, arXiv 2505.12705).
• Cosmos-Predict2.5 and Cosmos-Transfer2.5, the open foundation for most of this chapter (NVIDIA et al. 2025, arXiv 2511.00062).
• DreamerV3, for the idea of learning in imagination, before robots and video (Hafner et al. 2023, arXiv 2301.04104).
• World-VLA-Loop, for closing the loop between world model and policy (Liu et al. 2026, arXiv 2602.06508).
• WorldArena, for evaluating world models by what they are for (Shang et al. 2026, arXiv 2602.08971).
• EVA, for the executability gap (Wang et al. 2026, arXiv 2603.17808).
• Interactive World Simulator, for policy evaluation in a learned model (Wang et al. 2026, arXiv 2603.08546).