What is a robot foundation model

Why this chapter exists

"Robot foundation model" is used for almost anything a robotics company ships in 2026. Vendors apply it to vision-language-action policies, to video generators, to reasoning models that never touch a motor, and sometimes to a fine-tuned version of last year's product. A term that covers everything explains nothing, so this chapter pins it down: precisely enough to be useful, loosely enough to survive the next twelve months of releases.

The definition matters for the rest of the book in a practical way. The seven families of Part II are seven answers to two questions about a model: what does it predict, and in what space? If you can answer those from an abstract, you can place a new model within a minute and know which chapter explains how it works and where it breaks.

After this chapter you will be able to:

• say what makes a model a robot foundation model, and name three things that are useful to roboticists but are not;
• state the minimum interface every such model shares, and the one optional output that separates reactive models from world-action models;
• place a new model on the family grid from its abstract, and recognise the cases that sit on a boundary.

A working definition

Foundation models, in the sense the robotics surveys borrow from language and vision, are large networks trained on massive, heterogeneous data that are then adapted to many downstream tasks (Psiris et al. 2026, arXiv 2604.15395). A 2023 survey put the robotics version of the question directly: how can existing foundation models from language and vision be used for general-purpose robots, and what would a robotics-specific foundation model look like (Hu et al. 2023, arXiv 2312.08782)? Three years later the second question has dozens of answers.

This book uses a definition with three tests. A robot foundation model is a model that:

1. produces something a robot can act on: an action chunk, a prediction of the future that is used for control, or a plan that a controller below it executes;
2. was pretrained broadly: across many tasks, many robot bodies (embodiments), or web-scale video and images, rather than on one task in one lab;
3. adapts cheaply: by prompting, by conditioning on a goal, or by fine-tuning on a few hundred demonstrations rather than retraining.

The tests are deliberately about function, not architecture or size. They admit SmolVLA, a policy designed to be trained on a single GPU (Shukor et al. 2025, arXiv 2506.01844), and exclude the largest chat model that cannot move an arm.

The minimum interface

Every model in the book, whatever its internals, presents the same interface to the robot: observations and language in, an action chunk out. Observations are recent camera frames plus the robot's own joint state; the instruction is a sentence; the output is $H$ actions to execute at a fixed control rate. One inference produces many motor commands, which is what makes slow models usable on fast robots (chapter 2 shows the arithmetic).

The one optional output is the future. A world-action model also returns what it expects to see. That single addition is the line between chapter 5's reactive models and chapter 6's world-action models, and it is why the grid has a column for it.

In code the interface is one required method and one optional one. The chapter's Rust crate states it as a trait, and every later chapter's code implements some version of it:

/// What the robot senses at one instant: camera frames and its own joint state.
#[derive(Clone, Debug, Default)]
pub struct Observation {
    pub frames: Vec<Vec<u8>>,
    pub proprio: Vec<f32>,
}

/// H actions to execute at a fixed control rate. One inference, many motor commands.
#[derive(Clone, Debug, PartialEq)]
pub struct ActionChunk {
    pub rate_hz: f32,
    pub actions: Vec<Vec<f32>>,
}

/// The minimum interface of a robot foundation model: observations and language in,
/// an action chunk out. Predicting the future is optional, which is exactly the line
/// between reactive models and world-action models.
pub trait RobotFoundationModel {
    fn act(&self, history: &[Observation], instruction: &str) -> ActionChunk;

    /// Imagined future observations, if the model makes them.
    fn imagine(&self, _history: &[Observation], _instruction: &str) -> Option<Vec<Observation>> {
        None
    }
}

What the definition leaves out

Three neighbours are close enough to confuse and different enough to keep out.

A task-specific policy can be excellent: a diffusion policy trained on two hundred demonstrations of one task can be more reliable at that task than any generalist. It fails the second test. It has no breadth to adapt from, and a new task means new data and new training.

A classical modular pipeline splits perception, state estimation, planning and control into separate blocks, each designed and verified on its own. Its strengths, debuggability and certifiability, are exactly what learned models gave up (chapter 3 tells the story, chapter 15 counts the cost). It fails the first test as a whole: no single learned model produces the action.

A plain vision-language model answers questions about images. It has the breadth and adapts by prompting, but produces nothing a robot acts on. Fine-tuned to output actions it becomes the backbone of a chapter 5 model; trained for spatial reasoning and success detection it becomes a chapter 10 model. Off the shelf it is neither.

The industrial survey of March 2026 shows how large the category already is under a definition like this one: it evaluates 324 manipulation-capable robot foundation models against 149 readiness criteria (Kube et al. 2026, arXiv 2603.06749). Its verdict is the useful corrective to vendor language. Industrial maturity is limited and uneven, and even the highest-rated models satisfy only a fraction of the criteria.

Two questions that sort everything

The family grid of figure 1.1 asks two questions of every model.

What does it predict? Actions only, as in a reactive vision-language-action model (VLA); actions together with predicted futures, as in a world-action model (WAM); futures only, as in a world model used to plan or to generate data; or a plan in language, as in an embodied reasoning model.

In what space? Raw pixels or the latents of a video model; learned feature latents that are never decoded back to pixels (the JEPA approach of chapter 7); explicit geometry such as point clouds, Gaussians or occupancy grids (chapter 9); language or action tokens; or continuous actions with no language backbone at all (chapter 11's large behavior models).

The answers decide almost everything that matters in practice: which data a model can learn from, what it costs to run, and how it fails. A model that predicts in pixels can learn from internet video; a model that predicts only actions needs action-labelled robot data; a model that predicts in geometry needs depth or point clouds. Chapter 13's data pyramid makes this precise.

The chapter's crate turns the two questions into a function. A model card records what a model outputs and in what space; placing it on the grid is then mechanical, and exercise 1.1 asks you to write it:

/// Down the family grid: the space a model predicts in.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Space {
    PixelsOrVideoLatents,
    FeatureLatents,
    ExplicitGeometry,
    LanguageOrActionTokens,
    ContinuousNoLanguage,
}

/// Across the family grid: what a model predicts.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Predicts {
    ActionsOnly,
    ActionsPlusFutures,
    FuturesOnly,
    PlanInLanguage,
}

/// The facts about a model that decide where it goes, read off its paper or model card.
#[derive(Clone, Debug)]
pub struct ModelCard {
    pub name: &'static str,
    pub outputs_actions: bool,
    pub outputs_futures: bool,
    pub outputs_language_plan: bool,
    pub space: Space,
    /// Pretrained across many tasks, embodiments or web-scale data rather than one task.
    pub broad_pretraining: bool,
    /// Adapted to new tasks by prompting or light fine-tuning.
    pub adapts_cheaply: bool,
    /// Trained on embodied data (robot or egocentric), not only web text and images.
    pub embodied_training: bool,
}

Three boundary cases

The grid is approximate on purpose. Three models show how to handle a model that fits two cells.

π0.7 conditions on goal images as well as language: you can show it what the end of the task should look like (Physical Intelligence et al. 2026, arXiv 2604.15483). The WAM survey reads it as a goal-conditioned world-action model for exactly that reason (Lu et al. 2026, arXiv 2609.16074). This book places it with the reactive models of chapter 5, because at inference it never generates a future: the goal image is an input, not a prediction. The rule the book uses for boundary cases is to place a model by what it predicts at inference, and to say so when training tells a different story.

Octo is an open generalist policy trained on 800,000 trajectories from Open X-Embodiment and instructed by language commands or goal images (Octo Model Team et al. 2024, arXiv 2405.12213). It could sit with the VLAs of chapter 5 or the large behavior models of chapter 11. Its language encoder is small and its strength is the scaled imitation recipe, so chapter 11 discusses it as the bridge between the two.

GR00T N1.7 is a VLA by the grid's first reading, built on a Cosmos-Reason2 backbone with a diffusion transformer action head. For whole-body humanoid control it emits compact latent action tokens that a separate learned controller expands into joint commands (NVIDIA 2026). That path is a latent-action interface, chapter 7's territory. One product, two families, depending on the robot it drives.

How the families map to the surveys

The seven families are this book's synthesis, not a published taxonomy, and a reader who knows the surveys should be able to move between them. The WAM survey of September 2026 builds world-action models from four research threads: model-based reinforcement learning and world models, video generative models, latent action pretraining, and vision-language-action models (Lu et al. 2026, arXiv 2609.16074). The April 2026 comprehensive review organises the wider field by model type (language, vision, vision-language and vision-language-action models), architecture, learning paradigm and application (Psiris et al. 2026, arXiv 2604.15395). A June 2026 survey of world-action models offers two further views of the same models (Shen et al. 2026, arXiv 2606.20781).

Exercises

The stubs are in rust/ch01-interface/src/exercises.rs; reference solutions compile with --features solutions.

cargo test -p ch01-interface --test exercises ex1_1

cargo test -p ch01-interface --test exercises ex1_2

cargo test -p ch01-interface --test exercises ex1_3

Further reading

• The April 2026 comprehensive review, for the broadest map of the field and its five research phases (Psiris et al. 2026, arXiv 2604.15395).
• The 2023 survey and meta-analysis that first asked what a robotics-specific foundation model would look like (Hu et al. 2023, arXiv 2312.08782).
• The industrial readiness survey, for how far the category still is from the factory floor (Kube et al. 2026, arXiv 2603.06749).
• A 2025 anatomy of vision-language-action models, from modules to milestones (Xu et al. 2025, arXiv 2512.11362).
• Two earlier surveys of foundation models for manipulation and robot learning (Li et al. 2024, arXiv 2404.18201; Xiao et al. 2023, arXiv 2311.14379).