Latent prediction and latent action models

Why this chapter exists

Chapter 6 showed what a robot gains by predicting the future, and what it pays: generating video is expensive, and most of what a video model predicts is irrelevant to the robot. The exact texture of a tablecloth, the flicker of a screen, a shadow moving across the counter all cost model capacity and compute, and none of them change what the gripper should do.

This chapter covers two related ideas that answer that cost. The first is to predict in representation space: forecast the features of the next frame, not its pixels, so the model is free to ignore whatever it cannot or need not predict. The second is to infer actions that were never recorded. A model watching two consecutive frames of a person stirring a pot can compress "what happened" into a small latent action, and that turns the largest layer of chapter 2's data pyramid, video with no actions at all, into something a policy can learn from.

After this chapter you will be able to:

• write down the difference between a pixel loss and a latent loss, explain what each term does, and use figure 7.1 to say why the latent one learns faster from noisy footage;
• explain with figure 7.2 how a latent action model reads an action off two frames, and why a human hand and a robot gripper can share one;
• judge where a latent model's claims are strong (cheap inference, learning from human video) and where they are still weak (per-robot decoding, collapse, real-robot comparisons).

Predicting in the right space

Two losses

In chapter 4's notation, a model that predicts the next observation in pixels is trained to make its prediction $\hat o_{t+1}$ match the real frame:

Every pixel counts equally. If the background flickers in a way no model could foresee, that flicker shows up in the loss on every training step, and it is the same size whether the model got the action right or wrong.

A joint-embedding predictive architecture, or JEPA, moves the comparison into feature space (Assran et al. 2025, arXiv 2506.09985; Balestriero et al. 2025, arXiv 2511.08544):

Read it term by term. $f_\theta$ is an encoder that turns the current frame into a feature vector. $g_\phi$ is a predictor that takes those features and the action $a_t$ and guesses the next frame's features. $\bar f$ is a target encoder: it turns the real next frame into the features the guess is compared with. In most JEPAs it is a slowly updated copy of $f_\theta$ that receives no gradient. The loss never touches a pixel. Because the encoder is learned too, it is free to drop anything that cannot be predicted, which is the property JEPA-VLA's authors single out: predictive video embeddings are "adept at flexibly discarding unpredictable environment factors" (Miao et al. 2026, arXiv 2602.11832).

Figure 7.1 makes the difference concrete with an idealised encoder that keeps only where the block is. With no flicker, the whole pixel loss is about the action. At a flicker of 0.3, only about 61 percent of it is, and at 0.5 only about 36 percent: the rest is noise that pushes the model around on every step without teaching it anything. The feature loss stays entirely about the action. Exercise 7.2 reproduces the numbers.

The catch: collapse

The latent loss has an obvious cheat. If the encoder outputs the same vector for every frame, the predictor can match it perfectly and the loss is zero. This is representation collapse, and every JEPA needs a guard against it. The classic guards are the slowly updated target encoder and a stop-gradient. LeJEPA replaces these heuristics with a single regulariser, SIGReg, that pushes the embeddings towards an isotropic Gaussian distribution, which its authors prove is the best target for downstream prediction (Balestriero et al. 2025, arXiv 2511.08544). Exercise 7.1 builds a detector for the same failure in a latent action codebook.

What the family has shown

The lineage runs from general video features to robot control. DINO-WM predicts future DINOv2 patch features from offline trajectories and plans by optimising actions towards a goal image's features, solving tasks zero-shot in six environments (Zhou et al. 2024, arXiv 2411.04983). V-JEPA 2 was pretrained on over one million hours of internet video. Its action-conditioned variant, V-JEPA 2-AC, was post-trained on less than 62 hours of unlabelled robot video from DROID, then deployed zero-shot on Franka arms in two labs, picking and placing objects by planning towards image goals (Assran et al. 2025, arXiv 2506.09985). SkyJEPA took the idea to quadrotors, combining a JEPA dynamics model with sampling-based control on embedded hardware (Rao et al. 2026, arXiv 2606.23444).

In 2026 the idea reached VLAs and world-action models. VLA-JEPA pretrains with "leakage-free state prediction": future frames only produce targets, never inputs, so the model cannot learn to copy the answer (Sun et al. 2026, arXiv 2602.10098). JEPA-VLA adds V-JEPA 2 embeddings to existing VLAs and reports gains on LIBERO, LIBERO-Plus, RoboTwin 2.0 and real robots (Miao et al. 2026, arXiv 2602.11832). LaWAM predicts latent visual subgoals instead of future video. Its authors report 98.6 percent success on LIBERO, 187 ms per action chunk, and up to 24 times lower latency than pixel-space world-action models (Chen et al. 2026, arXiv 2606.15768). Two different August 2026 papers are both called JEPA-WAM. One couples latent transition prediction with action generation in a pretrained V-JEPA space; it reports 79.2 percent on LIBERO-Plus without large-scale robot pretraining and 86.3 percent when built on π0.5 (Lin et al. 2026, arXiv 2608.09381). The other encodes generated goal images with a frozen V-JEPA 2.1 encoder to improve instruction following (Liu et al. 2026, arXiv 2609.20277). Cite them by arXiv number.

Inferring actions that were never recorded

A latent action

Video of people doing things is abundant; robot actions are not. A latent action model bridges the two by learning, from video alone, a small code for "what changed between these two frames". It has two parts trained together: an inverse model $E$ that looks at both frames and outputs a code, and a forward model $D$ that must reconstruct the second frame from the first plus that code:

The trick is the bottleneck $q$. In LAPA it is a VQ-VAE codebook: the code is snapped to the nearest of a small set of learned vectors (Ye et al. 2024, arXiv 2410.11758). Because $z_t$ is tiny and the forward model already sees $o_t$, the cheapest thing for $z_t$ to carry is the change itself, the motion, not the appearance. Genie used this to make an 11-billion-parameter model of interactive environments from unlabelled internet video (Bruce et al. 2024, arXiv 2402.15391); robotics borrowed it to make video look like demonstration data.

The toy idealises the hard part: here the latent action model is told where the hand is. A real one must find the hand, ignore the camera shake and not smuggle appearance into the code. VLA-JEPA's authors argue that many latent-action objectives fail exactly here, staying "anchored to pixel variation rather than action-relevant state transitions" and vulnerable to "appearance bias, nuisance motion, and information leakage" (Sun et al. 2026, arXiv 2602.10098). ViPRA adds perceptual losses and optical-flow consistency to keep its latent actions physically grounded (Routray et al. 2025, arXiv 2511.07732). DreamDojo uses continuous rather than discrete latent actions as unified proxy actions across 44,000 hours of egocentric human video (Gao et al. 2026, arXiv 2602.06949).

From human video to one robot

A latent action says "the hand moved right" in the camera's view. A robot needs "joint 3 by 0.1 radians". The last stage learns that mapping from a small set of real robot demonstrations, labelled with both:

That is a far smaller problem than learning control from scratch, and the published numbers reflect it. ViPRA's decoder needs 100 to 200 teleoperated demonstrations and controls at up to 22 Hz; its authors report a 16 percent gain on the SIMPLER benchmark and 13 percent on real-world tasks over strong baselines (Routray et al. 2025, arXiv 2511.07732). LAPA reports outperforming a state-of-the-art VLA trained with action labels on real-world language-conditioned tasks, and positive transfer when pretraining on human manipulation video only (Ye et al. 2024, arXiv 2410.11758). Exercise 7.3 fits the toy decoder.

Latent tokens for a whole body

The same idea works one level down. GR00T N1.7 can drive a Unitree G1 humanoid through SONIC: the VLA "predicts compact latent action tokens that a learned whole-body controller decodes into full-body joint commands", including legs, arms and hands (NVIDIA 2026). SONIC itself is a motion-tracking foundation model scaled from 1.2 to 42 million parameters on more than 100 million frames from 700 hours of motion capture; a unified token space lets the same policy take commands from VR teleoperation or from a VLA (Luo et al. 2025, arXiv 2511.07820).

Here the latent space is not discovered from video but designed as an interface. It does the same job: it separates the part that needs language and vision from the part that needs physics, so each can be trained on the data that suits it.

The family on the four-component template

Why it exists, good at, bad at

Why it exists. Pixel prediction wastes capacity on appearance and can take shortcuts through it; latent targets force the model to learn the transitions that matter for control. Latent actions unlock video with no actions, the largest layer of the data pyramid.

What it is good at. Inference is cheaper than video generation, since nothing is rendered: LaWAM's reported 24-fold latency advantage over pixel-space WAMs is the clearest number (Chen et al. 2026, arXiv 2606.15768). Latent actions are the most direct bridge from human video to robot control, and they let one policy serve several bodies through per-robot decoders.

What it is bad at. Latent targets are hard to inspect. A video prediction can be watched; a feature vector cannot, so debugging a latent model means training probes. Collapse needs active prevention. And the last step, from latent action to real command, is still trained per embodiment on real robot data.

In code

The crate's strip world is small enough to read in one go: a block, a flickering background, a best-possible pixel predictor and an idealised encoder.

/// A one-dimensional camera image of `WIDTH` pixels: a bright block of `BLOCK` pixels that the
/// robot pushes, over a background that flickers unpredictably (lighting, shadows, a screen).
pub const WIDTH: usize = 48;
pub const BLOCK: f64 = 6.0;

/// Deterministic noise in [-1, 1] for pixel `i` of frame `seed`.
pub fn noise(seed: u64, i: usize) -> f64 {
    let mut x = seed.wrapping_mul(0x9E37_79B9_7F4A_7C15) ^ (i as u64).wrapping_mul(0xBF58_476D_1CE4_E5B9);
    x ^= x >> 31;
    x = x.wrapping_mul(0x94D0_49BB_1331_11EB);
    x ^= x >> 29;
    (x % 20_001) as f64 / 10_000.0 - 1.0
}

/// Render the strip with the block's left edge at `pos` (sub-pixel positions blend edge pixels).
pub fn render(pos: f64, flicker: f64, seed: u64) -> Vec<f64> {
    (0..WIDTH)
        .map(|i| {
            let (l, r) = (i as f64, i as f64 + 1.0);
            let cover = (r.min(pos + BLOCK) - l.max(pos)).clamp(0.0, 1.0);
            cover + (1.0 - cover) * flicker * noise(seed, i)
        })
        .collect()
}

/// The best a pixel predictor can do: move the block, and predict the background's average (zero),
/// because the flicker in the next frame cannot be known in advance.
pub fn predict_pixels(pos: f64, action: f64) -> Vec<f64> {
    render(pos + action, 0.0, 0)
}

/// An idealised feature encoder: where the block is. A JEPA-style encoder trained on footage like
/// this learns something close to it, because the flicker cannot be predicted and so is not worth
/// encoding. Here it is a matched filter: the window of `BLOCK` pixels with the most brightness.
pub fn encode(frame: &[f64]) -> f64 {
    let w = BLOCK as usize;
    (0..=WIDTH - w)
        .map(|s| (s, frame[s..s + w].iter().sum::<f64>()))
        .fold((0, f64::MIN), |best, cur| if cur.1 > best.1 { cur } else { best })
        .0 as f64
}

/// Mean squared error between two frames.
pub fn pixel_loss(pred: &[f64], actual: &[f64]) -> f64 {
    pred.iter().zip(actual).map(|(p, a)| (p - a).powi(2)).sum::<f64>() / pred.len() as f64
}

The latent action side is a codebook and a displacement. Everything hard about real latent action models is hidden in displacement, which here is given rather than learned.

/// A codebook of latent actions, as in a VQ-VAE latent action model: a handful of vectors, each
/// standing for one kind of change between two frames. Here: "stay", plus `k` directions.
#[derive(Clone, Debug)]
pub struct Codebook {
    pub codes: Vec<[f64; 2]>,
}

impl Codebook {
    pub fn ring(k: usize, step: f64) -> Codebook {
        let mut codes = vec![[0.0, 0.0]];
        codes.extend((0..k).map(|j| {
            let a = std::f64::consts::TAU * j as f64 / k as f64;
            [step * a.cos(), step * a.sin()]
        }));
        Codebook { codes }
    }
}

/// The idealised latent-action encoder: how far the tracked hand or gripper moved between two
/// frames. A learned latent action model has to discover this from pixels alone.
pub fn displacement(before: [f64; 2], after: [f64; 2]) -> [f64; 2] {
    [after[0] - before[0], after[1] - before[1]]
}

Exercises

The stubs are in rust/ch07-latent/src/exercises.rs.

cargo test -p ch07-latent --test exercises ex7_1

cargo test -p ch07-latent --test exercises ex7_2

cargo test -p ch07-latent --test exercises ex7_3

Further reading

• V-JEPA 2, for the clearest demonstration of planning in feature space on real robots (Assran et al. 2025, arXiv 2506.09985).
• DINO-WM, a compact and readable latent world model (Zhou et al. 2024, arXiv 2411.04983).
• LAPA, the paper that made latent actions a VLA pretraining recipe (Ye et al. 2024, arXiv 2410.11758).
• VLA-JEPA, for the argument against pixel-anchored latent actions (Sun et al. 2026, arXiv 2602.10098).
• LaWAM, for latent subgoals in a world-action model (Chen et al. 2026, arXiv 2606.15768).
• LeJEPA, for a principled answer to collapse (Balestriero et al. 2025, arXiv 2511.08544).
• SONIC, for latent tokens as an interface to a whole-body controller (Luo et al. 2025, arXiv 2511.07820).