Why now: the convergence

Why this chapter exists

Every idea in this book is older than the models that made it famous. Learning robot policies from demonstrations, predicting video, discretising actions into tokens, using a world model to plan: each has a paper from the 2010s or earlier. So why did robot foundation models arrive in 2024 to 2026 and not five years before?

The honest answer is that no one thing changed. Six ingredients matured within roughly the same 24 months: vision-language models good enough to be a backbone, video models good enough to be a dynamics prior, robot datasets large enough to pool, human video usable as robot data, inference fast enough for a robot, and economics that made fine-tuning cheaper than starting over. Each was necessary, none was sufficient. This chapter tells the story of their convergence, because that story predicts what the next convergence will need.

After this chapter you will be able to:

• name the six ingredients and say which family of model depends most on each;
• explain, with the simulator in figure 2.3, how action chunking turns a slow model into a fast controller;
• tell which of the commonly quoted numbers about this shift are measured, and which come from vendors or industry surveys.

1. Vision-language models became good enough to be a backbone

The first generalist robot policies borrowed their understanding of the world from vision-language models (VLMs). The recipe, which chapter 5 explains in full, is to take a pretrained VLM and teach it to emit actions. It only works if the VLM already knows what a mug is, where the handle sits and what "put it in the sink" means.

By 2024 open VLMs crossed that bar. PaliGemma, a 3-billion-parameter model built for transfer, became the backbone of π0 and π0.5 (Beyer et al. 2024, arXiv 2407.07726; Physical Intelligence et al. 2025, arXiv 2504.16054). Qwen-VL and its successors became backbones for others (Bai et al. 2023, arXiv 2308.12966). In 2026 π0.7 moved to Gemma 3 (Gemma Team et al. 2025, arXiv 2503.19786; Physical Intelligence et al. 2026, arXiv 2604.15483), and NVIDIA's GR00T N1.7 to Cosmos-Reason2, a VLM trained for physical reasoning and released in December 2025 (NVIDIA 2026). A backbone is now a choice, not a research project.

2. Video models became good enough to be a dynamics prior

A video generator that has watched enough of the world has learned, implicitly, how things move: what happens when a cup is tipped, how cloth falls. Chapter 6 showed what that buys a robot policy. The ingredient arrived in early 2025, when large open video models appeared. NVIDIA's Cosmos world foundation models were trained on about 100 million clips drawn from roughly 20 million hours of raw video (NVIDIA et al. 2025, arXiv 2501.03575). Wan's 14-billion-parameter model was trained on billions of images and videos (Team Wan et al. 2025, arXiv 2503.20314). A Google DeepMind study then showed a frontier video model solving visual tasks it had never been trained for (Wiedemer et al. 2025, arXiv 2509.20328).

The important word is open. When the video backbone had to be pretrained from scratch, the approach was out of reach for most robotics labs; once open models existed, a lab could fine-tune one (NVIDIA Technical Blog 2026).

3. Robot datasets crossed the million-trajectory mark

Robots produce data slowly. A trajectory is one demonstration of one task: minutes of an operator's time, recorded on one robot in one room. For years every lab trained on its own few thousand.

Pooling changed the scale. Open X-Embodiment gathered over one million real robot trajectories from 22 robot types, contributed by 21 institutions, in standard formats (Open X-Embodiment Collaboration et al. 2023, arXiv 2310.08864). DROID collected 76,000 trajectories (350 hours) across 564 scenes in the wild (Khazatsky et al. 2024, arXiv 2403.12945), and BridgeData V2 60,096 trajectories on a low-cost robot (Walke et al. 2023, arXiv 2308.12952). A million trajectories is still tiny next to the web, which is why the next ingredient mattered.

4. Human video became usable as robot data

There is far more video of people doing things with their hands than of robots doing anything. The problem is that human video carries no robot actions. Two developments made it usable. Better capture added the missing signal: EgoDex recorded 829 hours of egocentric video with tracked 3D hand and finger poses (Hoque et al. 2025, arXiv 2505.11709), and the UMI hand-held gripper turned human demonstrations into robot-ready trajectories (Chi et al. 2024, arXiv 2402.10329). Better learning made the rest of it usable: latent actions, chapter 7's subject, infer an action-like signal from consecutive frames.

The scale then moved quickly. EgoScale trained on 20,854 hours of action-labelled egocentric video and reports a log-linear scaling law between hours of human data and prediction loss (Zheng et al. 2026, arXiv 2602.16710). DreamDojo pretrained a world model on 44,000 hours (Gao et al. 2026, arXiv 2602.06949). In August 2026 Dyna Robotics reported pretraining on more than one million hoursvendor claim (Dyna Robotics 2026). The direction is clear. Whether the scaling holds across labs and robots is not yet established, and chapter 13 looks at the evidence.

5. Inference became fast enough for a robot

A robot arm runs its controller tens to hundreds of times a second. A large model takes a tenth of a second or more to think. Two ideas closed the gap.

The first is the action chunk: predict the next several actions at once, then execute them while the model is idle. ACT introduced the idea for robot imitation in 2023 (Zhao et al. 2023, arXiv 2304.13705). One inference now pays for many motor commands. The second is thinking while moving: compute the next chunk while the current one executes, so the robot never stops. Real-time chunking made this work for flow-based VLAs without retraining (Black et al. 2025, arXiv 2506.07339).

The simulator below shows why both matter. Set a 200-millisecond model and a 50 Hz controller. With one action per inference the robot delivers under 5 commands a second and spends most of its time waiting. Raise the chunk length and the waiting is amortised; switch to thinking while moving and the robot runs at its controller's full rate as soon as a chunk lasts longer than an inference.

Hardware and engineering did the rest. Quantised models, exported to inference engines and run on robot-mounted computers such as NVIDIA's Jetson Thor (NVIDIA Newsroom 2026), now reach control-relevant rates on the robot itself. GR00T N1.7 ships with ONNX and TensorRT export (NVIDIA Technical Blog 2026). An industry survey puts quantised VLAs at 10 to 25 Hz on consumer-grade GPUsindustry survey (SVRC (Robotics Center of Silicon Valley) 2026), and a 2026 study across edge accelerators finds that right-sized devices can meet control-rate constraints at lower cost and energy than flagship GPUs (Zhou et al. 2026, arXiv 2604.24447).

/// Synchronous execution: the robot stops while the model thinks, then runs the whole chunk.
/// Returns motor commands per second actually delivered.
pub fn sync_rate_hz(latency_ms: f32, chunk_len: usize, control_hz: f32) -> f32 {
    let execute_ms = chunk_len as f32 * 1000.0 / control_hz;
    chunk_len as f32 * 1000.0 / (latency_ms + execute_ms)
}

/// Asynchronous execution: the next chunk is computed while the current one runs. The robot never
/// stops as long as inference finishes before the steps it is executing run out.
pub fn async_keeps_up(latency_ms: f32, execute_steps: usize, control_hz: f32) -> bool {
    latency_ms <= execute_steps as f32 * 1000.0 / control_hz
}

6. The economics flipped

The last ingredient is money. For most of the 2010s a new robot task meant collecting thousands of demonstrations and training a policy from scratch. Two numbers from the State of Robotics 2026 report describe the change, and both should be read as industry-survey claims rather than measurements.

The first is the price of data. One hour of teleoperation data, captured, labelled and packaged, fell from about $340 in early 2024 to$136 by the end of 2025 and $118 by March 2026industry survey (SVRC (Robotics Center of Silicon Valley) 2026).

The second is how much data a new task needs. The same report finds that fine-tuning a pretrained VLA on 200 to 500 demonstrations now outperforms a task-specific policy trained from scratch on more than 1,000industry survey. Model makers report smaller numbers still, for their own models: LingBot-VLA used 130 post-training episodes per task in its evaluation (Wu et al. 2026, arXiv 2601.18692), and MotuBrain adapts to a new humanoid from 50 to 100 trajectoriesvendor claim (Motubrain Team et al. 2026, arXiv 2604.27792).

When fine-tuning a shared base beats building from scratch, the economically rational move is to build the base. That is the moment foundation models stop being a research bet and become a product strategy, which is how chapter 3's fourth phase begins.

What converged, and what it predicts

Laid side by side, the six tracks in figure 2.1 explain the timing better than any single milestone. The robot datasets of 2023 had nothing strong enough to learn from them until the VLM backbones of 2024. The video models of 2025 had no way to drive a robot until action chunking and fast inference made the output usable. Human video was a curiosity until the latent-action and hand-tracking work of 2025 and 2026 gave it a robot-shaped signal.

The pattern suggests what to watch. The next shift will need its own convergence: probably cheap, broad real-robot evaluation (chapter 15), safety cases for learned controllers (chapter 17), and memory for tasks longer than a few minutes (chapter 14). Each is still a single track.

The chapter's Rust crate also fits the one scaling curve in this chapter with a stated number. EgoScale reports average task completion rising from 0.30 after 1,000 hours of human pretraining to 0.71 after 20,000 (Zheng et al. 2026, arXiv 2602.16710). A log-linear curve through those two points is easy to draw, and exercise 2.3 shows how quickly it starts promising more than 100 percent task completion if you follow it beyond the data.

/// A log-linear scaling curve, y = a + b ln(x), the form EgoScale reports for human-video hours.
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct LogLinear {
    pub a: f64,
    pub b: f64,
}

impl LogLinear {
    pub fn predict(&self, x: f64) -> f64 {
        self.a + self.b * x.ln()
    }
}

/// EgoScale reports average task completion of 0.30 after 1k hours of egocentric pretraining
/// and 0.71 after 20k hours (Zheng et al. 2026, arXiv 2602.16710, section 5).
pub const EGOSCALE_POINTS: [(f64, f64); 2] = [(1_000.0, 0.30), (20_000.0, 0.71)];

Exercises

The stubs are in rust/ch02-why-now/src/exercises.rs.

cargo test -p ch02-why-now --test exercises ex2_1

cargo test -p ch02-why-now --test exercises ex2_2

cargo test -p ch02-why-now --test exercises ex2_3

Further reading

• Open X-Embodiment, the dataset that made pooling robot data normal (Open X-Embodiment Collaboration et al. 2023, arXiv 2310.08864).
• EgoScale, for the strongest public case for human video at scale (Zheng et al. 2026, arXiv 2602.16710).
• Real-time chunking, for how asynchronous execution works with flow-based policies (Black et al. 2025, arXiv 2506.07339).
• The NVIDIA post on world-action models, for why open video backbones changed what a lab can afford (NVIDIA Technical Blog 2026).
• The State of Robotics 2026 report, read as an industry survey (SVRC (Robotics Center of Silicon Valley) 2026).