Choosing between families

Why this chapter exists

Part II described seven families one at a time. A practitioner does not meet them that way. They meet a job: a cell to automate, a robot to make useful, a research question to answer. They want to know which family to start with, what it will cost in data and compute, and how it will fail first.

This chapter adds no new models. It puts the seven families side by side, turns the comparison into a short sequence of questions, and walks three realistic jobs through it. Treat it as a map drawn on a particular date. The matrix encodes the authors' judgement as of October 2026, and some of it will be wrong within a year.

After this chapter you will be able to:

• read the comparison matrix of figure 12.1, including which cells rest on independent evidence and which on vendor claims;
• walk a new job through the decision flow of figure 12.2 and name a first family to try;
• predict the first failure mode for that choice, and what to measure to catch it.

Reading the matrix

Each column of figure 12.1 is a family and each row a criterion that a deployment team cares about. The scores have three levels, so that nobody mistakes them for measurements. Higher is always better, so "cheap inference" scores high for a small, fast model and low for one that generates video.

Three rows deserve a warning. Little robot data needed mixes two things: how much action-labelled data the family needs to pretrain, and how much a new task needs. Large behavior models need few demonstrations per task once pretrained, but every task must still be demonstrated, so the row scores them low. Open weights counts what can be downloaded today, not what is promised. Evidence quality is the row to read first. It says whether a family's other scores rest on shared benchmarks and independent evaluation, or mostly on what the makers report. Reactive VLAs and large behavior models score high there. World models used as simulators and embodied reasoning models score low, because most of what is known about them comes from vendor posts and surveys.

Two things are worth trying in the figure. First, with all weights equal, reactive VLAs come out on top, mostly on maturity, open weights and evidence. Second, the ranking is fragile. Raising the weight on reasoning about consequences from 1 to 2.25 hands first place to world-action models, level with world models used as simulators, and raising the weight on cheap inference by the same amount hands it to large behavior models. Exercise 12.3 measures how much each criterion must move to change the answer. One structural fact survives every weighting: no family is dominated, meaning none is matched or beaten on every criterion by another. That is the strongest argument that the seven families are real alternatives rather than a ranking in disguise. Exercise 12.2 checks it.

The decision walk-through

Figure 12.2 turns the matrix into four questions. They are ordered from the cheapest answer to the most expensive, so that a job stops at the simplest family that fits.

Is it a short-horizon task with objects like those in the training data? If yes, start with a reactive VLA. It is the most documented, best-supported and cheapest family to fine-tune, and chapter 5 showed that hundreds of demonstrations often suffice. Most industrial jobs that are automated today look like this.

Does it involve contact, occlusion, or effects that appear only later? If yes, a reactive policy will make locally sensible moves that ruin later steps, which is the gap chapter 6 opened with. Start with a world-action model, or a latent-prediction model if inference cost matters (chapter 7). If the contact state itself is what matters, as in insertion or cloth handling, add geometry-first models to the list (chapter 9).

Is there action-labelled robot data for this body? If no, the realistic route is to pretrain on video with latent actions and adapt with a small decoder (chapter 7), and to use a world model to multiply whatever data you collect (chapter 8).

Is the robot restricted to onboard compute? If yes, a quantised VLA on the robot, with an onboard reasoning model only if it fits. If a network is available, put an embodied reasoning model above a controller (chapter 10). The controller can be a VLA, or a large behavior model where the tasks are fixed and speed matters most (chapter 11).

The flow is deliberately simple, and real jobs blur its questions. A household task can be short-horizon in one room and long-horizon in the next. The questions are still the right ones to ask first, because each one corresponds to a property that separates the families: horizon, contact, data and compute.

Three worked scenarios

The three cards share a pattern worth making explicit. The first failure is rarely the model's headline weakness. For the warehouse VLA it is not reasoning about consequences, which the job does not need, but distribution shift in the products. For the household humanoid it is not the controller but the judgement of when a step is done. For the inspection robot it is perception and connectivity. In each case the useful question is not "which family is best?" but "how will this family fail on this job, and how will I know?"

In code

The decision flow fits in a dozen lines. Writing it as code makes its assumptions inspectable, and easy to argue with.

#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum Family {
    ReactiveVla,      // chapter 5
    WorldAction,      // chapter 6
    Latent,           // chapter 7
    Simulator,        // chapter 8
    Geometry,         // chapter 9
    Reasoning,        // chapter 10
    LargeBehavior,    // chapter 11
}

/// What you know about the job before choosing.
#[derive(Clone, Copy, Debug, Default)]
pub struct Job {
    pub short_horizon_in_distribution: bool,
    pub contact_occlusion_or_delayed_effects: bool,
    pub robot_action_data_for_this_body: bool,
    pub edge_compute_only: bool,
    pub precise_contact_state_matters: bool,
}

/// The decision flow of figure 12.2: a first family to try, and what to consider next.
pub fn recommend(job: &Job) -> Vec<Family> {
    use Family::*;
    if job.short_horizon_in_distribution {
        return vec![ReactiveVla];
    }
    if job.contact_occlusion_or_delayed_effects {
        let mut v = vec![WorldAction, Latent];
        if job.precise_contact_state_matters {
            v.push(Geometry);
        }
        return v;
    }
    if !job.robot_action_data_for_this_body {
        return vec![Latent, Simulator];
    }
    if job.edge_compute_only {
        return vec![ReactiveVla, Reasoning];
    }
    vec![Reasoning, ReactiveVla, LargeBehavior]
}

Exercises

The stubs are in rust/ch12-choosing/src/exercises.rs. This chapter is equation-free; the exercises are about weighing, comparing and stress-testing a judgement.

cargo test -p ch12-choosing --test exercises ex12_1

cargo test -p ch12-choosing --test exercises ex12_2

cargo test -p ch12-choosing --test exercises ex12_3

Further reading

• The industrial readiness survey, for a far more detailed set of criteria (Kube et al. 2026, arXiv 2603.06749).
• The WAM survey, for the taxonomy that most of Part II builds on (Lu et al. 2026, arXiv 2609.16074).
• The April 2026 comprehensive review of foundation models in robotics, for the broader landscape (Psiris et al. 2026, arXiv 2604.15395).
• TRI's careful examination of large behavior models, for how to evaluate a choice once made (TRI LBM Team et al. 2025, arXiv 2507.05331).
• Rodney Brooks' annual predictions scorecard, for a long view on hype and deployment (Rodney Brooks 2026).