Geometry-first and structured world models

Why this chapter exists

Chapters 6 to 8 predicted the future as video or as features learned from video. That choice inherits something valuable, the billions of hours of video on the web, and something awkward. A video frame is a picture of the world, not the world. It does not say how far away the mug is, whether the cloth is touching the box, or how hard the peg is pressing on the rim of the hole. For many tasks the picture is enough. For contact-rich manipulation it often is not, because the variables control depends on are exactly the ones a camera reports worst.

Geometry-first models predict in an explicit spatial or physical representation instead: point clouds, Gaussian splats, particles, occupancy grids, and in some cases touch or sound. They give up web-scale pretraining in exchange for states a physicist would recognise, predictions that can be checked against a ruler, and simulation that does not need to render a single image.

After this chapter you will be able to:

• name the main geometric representations, and say with figure 9.1 what each makes explicit and what it loses;
• explain with figure 9.2 why an RGB rollout can be underspecified for contact, using a concrete pixel-size argument;
• score a predicted point cloud with chamfer distance and voxel IoU, and say why a point-by-point loss would be wrong.

The mechanism

A geometry-first world model is a forward dynamics model whose state is geometry. Write the scene as a set of points $P_t$, each a 3D position, possibly with a colour or material label. The model predicts how that set moves under an action:

The same line covers Gaussians (each element has a position, a shape and a colour), particles (each element is a lump of material, joined to its neighbours) and occupancy grids (each element is a filled cell of space). What they share is that the state is a list of things in space, not a grid of colours. A prediction can then be checked with geometry: is the mug where the model said it would be, to the millimetre?

That shift has a practical consequence for the action too. PointWorld represents actions as "3D point flows": the predicted motion of points on the robot's own body, rather than joint angles. That lets a single model learn from a Franka arm and a bimanual humanoid without an embodiment-specific action space (Huang et al. 2026, arXiv 2601.03782).

What RGB misses

The case for geometry is clearest at contact. Figure 9.2 puts a peg over a hole with a millimetre of clearance and looks at it two ways.

With 2 mm pixels and 1 mm of clearance, every offset between 0.6 and 1.9 mm jams the peg on the rim, and every one of them produces exactly the same camera image as a perfect alignment. Exercise 9.1 counts them. Real cameras are better than 2 mm per pixel at close range, and multiple views help; but contact also happens where the gripper's own fingers hide it, and the quantity that matters, the force on the rim, is not visible at all. A world model trained only on pixels has nothing to learn the difference from.

Visuo-tactile world models make this point with data. VT-WM adds touch to vision and reports 33 percent better object permanence and 29 percent better compliance with the laws of motion in its imagined rollouts, the failure modes being "objects disappearing, teleporting, or moving in ways that violate basic physics" under occlusion. In zero-shot real-robot planning it reports up to 35 percent higher success, with the largest gains on multi-step, contact-rich tasks (Higuera et al. 2026, arXiv 2602.06001). Sound can play the same role where vision is ambiguous: an audio world model predicts the rising pitch of a bottle being filled (Zhang et al. 2025, arXiv 2512.08405).

The representation ladder

Point clouds are the most direct. PointWorld is a large pretrained 3D world model: from one or a few RGB-D images and a sequence of actions, it forecasts per-point 3D displacements. It was trained on about 2 million trajectories and 500 hours of real and simulated manipulation, and runs in real time at 0.1 s per prediction (Huang et al. 2026, arXiv 2601.03782). ParticleFormer is a transformer over point clouds that models rigid, deformable and flexible materials interacting, trained directly on real robot perception data without elaborate scene reconstruction, and used for model predictive control (Huang et al. 2025, arXiv 2506.23126).

Gaussian splats represent a scene as many small coloured blobs that can be rendered back into images, so the model can be trained with photographs and still have an explicit 3D state. GWM predicts how Gaussian primitives propagate under robot actions with a latent diffusion transformer, and serves both as a training signal for imitation and as a simulator for model-based RL (Lu et al. 2025, arXiv 2508.17600). ManiGaussian++ uses a hierarchical Gaussian world model for two-armed manipulation (Yu et al. 2025, arXiv 2506.19842). 4DGS-WAM, from August 2026, separates dynamic objects from the static background so that only the objects need to be predicted. Its first experiments are on a driving dataset, KITTI-MOT, a reminder of how early this line is (Ma et al. 2026, arXiv 2608.25956).

Particles and physics go furthest towards a simulator. PIN-WM identifies a 3D rigid-body system from visual observations through differentiable physics, learns from a few task-agnostic interactions, and bridges the sim-to-real gap with physics-aware "digital cousins" (Li et al. 2025, arXiv 2504.16693). PhysWorld reconstructs a physical world from a generated video and grounds the video's motion in physically accurate actions with residual RL, without collecting real robot data (Mao et al. 2025, arXiv 2511.07416).

Other channels. Cloth-WM feeds surface normals into a DreamerV2-style world model to unfold cloth in the air, deployed zero-shot on a real robot (Rome et al. 2026, arXiv 2602.16675). On the policy side, SpatialVLA injects 3D position information into a VLA pretrained on 1.1 million real robot episodes (Qu et al. 2025, arXiv 2501.15830), and GeoVLA adds a point-cloud encoder next to the VLM (Sun et al. 2025, arXiv 2508.09071). Those two are reactive VLAs with better eyes, not world models, and figure 9.5 places them accordingly.

Scoring a geometric prediction

A predicted video is scored pixel by pixel. A predicted point cloud cannot be, because its points come in no particular order: point 17 of the prediction has no reason to correspond to point 17 of the truth. The standard answer is the chamfer distance. For each predicted point, find the nearest real point; for each real point, the nearest predicted one; average both:

The first term punishes predicted points with nothing real nearby, such as a handle where there is none. The second punishes real points the prediction missed, such as a handle the model forgot. Figure 9.4 lets you see both, and shows what goes wrong with a point-by-point loss once the order is shuffled. ParticleFormer's "hybrid point cloud reconstruction loss" is built from terms of this kind (Huang et al. 2025, arXiv 2506.23126).

Occupancy is the coarse alternative: divide space into cells, mark each cell as filled or empty, and compare the sets with intersection over union. It forgives small errors inside a cell and punishes errors that cross cells, so a 10 mm grid scores a 5 mm shift almost as perfect (an IoU of 0.89 in the crate's mug) and a 10 mm shift as poor (0.25). Exercise 9.3 builds both.

The family on the four-component template

Good at, bad at

Why it exists. RGB rollouts are underspecified for contact-rich manipulation; geometry exposes the variables control depends on; and simulation without rendering is cheaper.

What it is good at. States are physically grounded and interpretable: a predicted point cloud can be measured. Geometry fits naturally with classical physics engines and with structured sensing such as depth cameras, LiDAR and tactile sensors. And it can be fast: PointWorld predicts in 0.1 s (Huang et al. 2026, arXiv 2601.03782).

What it is bad at. It loses most of the web-scale pretraining that video and RGB models inherit, because the internet has far less 3D and tactile data than video. It is sensor-specific: a model trained on one depth camera or one tactile skin does not transfer freely. And the research thread is young, with no shared benchmark suite.

In code

The peg-and-camera model behind figure 9.2 is three short methods.

/// A peg above a hole, seen by a camera whose pixels each cover `px_mm` of the scene. The peg is
/// `offset_mm` to the side of the hole's centre; the hole is `clearance_mm` wider than the peg.
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct PegInHole {
    pub offset_mm: f64,
    pub clearance_mm: f64,
    pub px_mm: f64,
}

impl PegInHole {
    /// The pixel column the peg's edge falls in: all a camera can report about the offset.
    pub fn edge_pixel(&self) -> i64 {
        (self.offset_mm / self.px_mm).floor() as i64
    }
    /// The peg jams on the rim when it is off-centre by more than half the clearance.
    pub fn jams(&self) -> bool {
        self.offset_mm.abs() > self.clearance_mm / 2.0
    }
    /// How far the peg overlaps the rim, which is what a contact or force sensor would feel.
    pub fn overlap_mm(&self) -> f64 {
        (self.offset_mm.abs() - self.clearance_mm / 2.0).max(0.0)
    }
}

Exercises

The stubs are in rust/ch09-geometry/src/exercises.rs.

cargo test -p ch09-geometry --test exercises ex9_1

cargo test -p ch09-geometry --test exercises ex9_2

cargo test -p ch09-geometry --test exercises ex9_3

Further reading

• PointWorld, the largest pretrained 3D world model so far (Huang et al. 2026, arXiv 2601.03782).
• ParticleFormer, for multi-material point-cloud dynamics (Huang et al. 2025, arXiv 2506.23126).
• GWM, for Gaussian world models as simulators (Lu et al. 2025, arXiv 2508.17600).
• PIN-WM, for physics-informed world models and digital cousins (Li et al. 2025, arXiv 2504.16693).
• VT-WM, for what touch adds to imagination (Higuera et al. 2026, arXiv 2602.06001).
• The WAM survey's section on structured world models (Lu et al. 2026, arXiv 2609.16074).