Start here
How to Read This Book
The organising idea, the reading paths, how evidence is marked, and what you need to run the Rust lab.
This book is organised so that you can read it three different ways, and all three are legitimate. Every chapter has the same shape, so once you know where things are in one chapter you know where they are in all nineteen.
The organising idea
The book sorts robot foundation models by two questions. What does the model predict: actions only, actions and futures, futures only, or a plan in language? And in what space does it predict: pixels, learned features, explicit geometry or tokens? The answers place every model on one grid, and the seven filled regions are the seven families of Part II. One colour per family, fixed across the book; colour in a figure means family and nothing else.
- Reactive VLAs
- World-action models
- Latent prediction
- World simulators
- Geometry-first
- Embodied reasoning
- Large behavior models
- Reactive vision-language-action modelsActions only
- World-action modelsActions plus futures
- Latent prediction and latent actionsFutures in feature space
- World models as simulatorsFutures only
- Geometry-first world modelsFutures as geometry
- Embodied reasoning modelsA plan in language
- Large behavior modelsActions, no language
Reading paths
- The executive path: Chapter openers and all Part II figures, then chapter 12, chapter 17 and chapter 19. About 25 pages.
- The engineer path: Everything, in order. The Rust exercises assume this path, and each chapter’s crate builds on the toolkit of chapter 4.
- The researcher path: Chapter 4, all of Part II, Part IV and the sources. About 90 pages.
What every chapter contains
In every chapter
- An opening figure that summarises the chapter before the text begins.
- Why it exists, and three things you will be able to do afterwards.
- A verbatim quote from a named person, linked to its source and dated.
- The body, with an interactive figure or simulation for each main idea.
- What we are not sure about: the honest concession, in every chapter without exception.
- Exercises in Rust, and further reading.
Math levels
L0No equations at all: intuition, diagrams and simulations.L1One boxed equation per idea, always explained in words and drawn.L2Equations with a term-by-term walkthrough. Chapters 4, 6 and 7 only.
How evidence is marked
Every claim is cited inline with its arXiv number or source, and each chapter ends with its full reference list. Numbers that do not come from independent measurement are tagged where they appear:
- A number a company reports about its own model or data vendor claim.
- A number from a market or practitioner survey industry survey.
- A position on a public leaderboard leaderboard claim.
Figures carry a one-sentence takeaway, a factual caption and a source. Figures that are not measured data say so: a schematic is the authors’ drawing of an idea, and a toy simulation is a small model built to show a mechanism, not to estimate a real number. Every chapter carries a “last checked” date, because the field moves monthly.
The data pyramid
The second idea that recurs through the book: most of what separates the families is which layer of data they can learn from. Robot trajectories are the smallest and most expensive layer, internet video the largest and least specific.
Robot trajectories
| Example | Reported size | Source |
|---|---|---|
| Open X-Embodiment | 1M+ trajectories, 22 embodiments | O'Neill et al. 2023, arXiv 2310.08864 |
| DROID | 76k trajectories, 350 hours, 564 scenes | Khazatsky et al. 2024, arXiv 2403.12945 |
| BridgeData V2 | 60,096 trajectories, 24 environments | Walke et al. 2023, arXiv 2308.12952 |
Who can use it: every family for fine-tuning; reactive VLAs (chapter 5) and large behavior models (chapter 11) learn mostly from this layer.
The Rust lab
Each chapter has a crate in rust/chNN-… with three exercises. Stubs are in src/exercises.rs; the tests in tests/exercises.rs fail until you implement them. Code shown on a page is read from the crate at build time, so what you read is what compiles. To check one exercise:
cargo test -p ch05-reactive-vla --test exercises ex5_1To run every chapter against the reference solutions, enable each crate’s solutions feature. You do not need to know Rust before starting; the exercises are small, and each comes with hints.
Ready? Start with chapter 1: What is a robot foundation model.