Open problems

Why this chapter exists

The previous sixteen chapters described what robot foundation models can do. This one collects what they cannot yet do, in the form the 2026 surveys converge on. The WAM survey names four major bottlenecks: action grounding, spatial consistency, closed-loop policy improvement and real-time inference (Lu et al. 2026, arXiv 2609.16074). The April 2026 comprehensive review and the industrial readiness survey add data, evaluation and safety (Psiris et al. 2026, arXiv 2604.15395; Kube et al. 2026, arXiv 2603.06749). The result is ten problems. Each is restated here in plain language, with one concrete failure taken from earlier in the book, so the problem is something you have seen rather than something you are told.

Two of the ten are under-researched relative to their importance: evaluation and safety. The source base for them is thin, and the chapter says so rather than padding.

After this chapter you will be able to:

• state each of the ten problems in a sentence and point to a failure that illustrates it;
• say which problems belong to particular families and which cut across all of them, using figure 17.1;
• explain why evaluation and safety sit in the severe but under-researched corner of figure 17.2.

How severe, how studied

The placement is a judgement, made from how much recent work in this book's sources addresses each problem. Latency, action alignment and neural simulation receive steady attention, with many 2026 papers on each. Evaluation and safety receive little. A handful of benchmark critiques and one industrial survey stand against a deployment record that grows every quarter.

1. Action alignment

A video or latent model learns how the world moves; a robot needs joint commands. Turning one into the other without destroying what the model learned is the first bottleneck the WAM survey names. Its recommendations are staged training that aligns predicted motion to executable actions, and protection of the pretrained prior through frozen backbones, alignment layers and residual action heads (Lu et al. 2026, arXiv 2609.16074). The failure is figure 8.4's: a generated rollout in which every frame looks right and the implied motion is impossible for the robot (Wang et al. 2026, arXiv 2603.17808).

2. World-action factorisation

Should futures and actions be learned together, and should futures still be generated at test time? Fast-WAM's answer is to train with video and drop it at inference (Yuan et al. 2026, arXiv 2603.16666). LingBot-VLA 2.0 arrives at a similar design from the VLA side, with future prediction as a training-only proxy task (Wu et al. 2026, arXiv 2607.06403). The failure is paying for video generation at every step, the cost curve of figure 6.6, for a benefit a training-only objective might have given. A March 2026 robustness study complicates the picture. It found world-action models robust under perturbation, with LingBot-VA reaching 74.2 percent on RoboTwin 2.0-Plus and Cosmos Policy 82.2 percent on LIBERO-Plus, while VLAs such as π0.5 matched them on some tasks only with extensive and varied training (Zhang et al. 2026, arXiv 2603.22078).

3. Spatial and multi-view consistency

Robots usually have several cameras, and a model that generates the future generates several views of it. If those views disagree about where an object is, the plan built on them is wrong in a way no single view reveals. The WAM survey lists multi-view consistency among its open challenges (Lu et al. 2026, arXiv 2609.16074), and VLA-MBPO needed a dedicated decoding scheme to enforce it (Zhang et al. 2026, arXiv 2603.20607). Figure 17.3 shows the check that exposes the problem.

The geometry is simple. Two bearings from cameras at known positions meet at one point $\hat p$. A third camera's predicted bearing $\theta_C$ should point at it, so the disagreement is

An error of 4 degrees in one generated view moves the located object by about 17 centimetres and makes the third view disagree by about 2 degrees: invisible frame by frame, obvious when compared. The failure generalises chapter 9's point: when futures are generated as images rather than as geometry, consistency is something to check, not something to assume.

4. Long-horizon memory

A policy that sees only the last moment forgets what it has done. MEM's combination of video short-term memory and text long-term memory reaches tasks of up to fifteen minutes (Torne et al. 2026, arXiv 2603.03596), and the WAM survey lists long-horizon memory among its open challenges (Lu et al. 2026, arXiv 2609.16074). The failure is figure 14.5's robot ten minutes into cooking that no longer knows whether the stove is on. The open question is architectural: memory inside the model, or an agentic harness around it that keeps state, as an embodied reasoning model does (chapter 10).

5. Neural simulators and closed-loop learning

Can a learned world model be trusted as a training environment? Errors in an imagined rollout compound: each predicted frame is the input to the next, so an error $e_t$ grows roughly as

where $\gamma$ is how much the model amplifies an existing error and $\varepsilon$ is the new error it adds each step. With $\gamma = 0.1$, thirty steps turn a per-step error of 0.01 into more than 1.6, while showing the model a real observation every five steps keeps it near 0.01. Exercise 17.1 computes both. The failure is figure 8.3's policy that finds the model's blind spot, and figure 6.4 shows the drift itself. WorldArena found that visual quality is a poor guide to whether a world model is useful for training or evaluation (Shang et al. 2026, arXiv 2602.08971).

6. Inference latency and compute

Large models are slow and robots are impatient. The WAM survey lists inference efficiency among its four bottlenecks (Lu et al. 2026, arXiv 2609.16074), and a cross-accelerator study found VLA inference split between a compute-bound backbone and a memory-bound action expert, wasting hardware (Zhou et al. 2026, arXiv 2604.24447). The failure is figure 15.1's robot that is blind for two seconds while finishing a chunk. This is the problem with the most mature solutions (chapter 18), but world-action models that generate video at test time make it worse again.

7. Data scarcity and provenance

The top of the pyramid, action-labelled robot data, stays small and expensive, and the reported hours of different sources are not a common currency (chapter 13). The failure is figure 13.3's two curves that fit the same scaling points and disagree completely beyond them, and the unsourced survey claim that 40 percent synthetic data matches real dataindustry survey (SVRC (Robotics Center of Silicon Valley) 2026).

8. Evaluation

The field's benchmarks are fragmented and mostly simulated, its main public real-robot leaderboard covers short tasks that evaluators choose, and there is no shared protocol for long tasks on real robots (chapter 15). The evidence that this matters is now direct. LIBERO-Plus found success falling from 95 percent to below 30 percent under modest perturbations, and models ignoring their instructions (Fei et al. 2025, arXiv 2510.13626). A study of the BEHAVIOR-1K challenge argued that final-state metrics exaggerate performance and hide safety violations (Rasouli et al. 2026, arXiv 2604.21192). BeTTER found state-of-the-art VLAs failing "catastrophically" in dynamic scenarios that static benchmarks never test (Xu et al. 2026, arXiv 2604.18000). The failure is figure 11.4's: with the 20 trials common in papers, a real 10-point improvement is detected less than one time in five.

9. Safety, verification and certification

Functional-safety standards such as ISO 10218 and ISO/TS 15066 assume components that can be specified and tested (Kube et al. 2026, arXiv 2603.06749). A learned policy cannot be, so today it sits outside the certifiable boundary, wrapped by an independent monitor (figure 15.5). The industrial survey found the highest-rated model fulfilling 6 percent of its safety and compliance criteria (Kube et al. 2026, arXiv 2603.06749). Google's model card for Gemini Robotics ER 2 rules out safety-critical uses (Google DeepMind 2026), and Brooks notes that no one has yet solved the humanoid version of the emergency stop (Rodney Brooks 2026).

The monitor's rule fits in one line:

where $v_t$ is the policy's command, $d_t$ the distance to the nearest person and $u_t$ what reaches the motors. A rule this simple can be certified; the policy it supervises cannot. The failure that remains is everything inside the envelope. A monitor stops the robot from moving too fast near a person. It does not stop the robot from carefully putting a knife in the wrong place.

10. Embodiment transfer and the humanoid question

Does whole-body control belong inside the foundation model, or in a learned controller beneath it? GR00T N1.7 answers with latent action tokens decoded by SONIC (NVIDIA 2026; Luo et al. 2025, arXiv 2511.07820), and Helix with a fast controller under a slow reasoner (Figure AI 2025). The failure is a model that transfers from one arm to another but must relearn everything for a new body, or that leaves balance to a layer the foundation model cannot see (figure 16.2).

In code

The two geometric helpers behind figure 17.3 are a bearing and the smallest difference between two angles.

pub type P = [f64; 2];

/// A camera on the floor plan: where it is. It reports only the bearing, in radians, from its
/// position to an object, which is what a generated image pins down well.
#[derive(Clone, Copy, Debug)]
pub struct Camera {
    pub at: P,
}

pub fn bearing(cam: Camera, p: P) -> f64 {
    (p[1] - cam.at[1]).atan2(p[0] - cam.at[0])
}

/// Three cameras watching a workspace, as on a bimanual rig with a head and two wrist views.
pub fn rig() -> [Camera; 3] {
    [Camera { at: [0.0, 0.0] }, Camera { at: [4.0, 0.0] }, Camera { at: [2.0, -1.5] }]
}

/// Smallest difference between two angles, in radians.
pub fn angle_diff(a: f64, b: f64) -> f64 {
    let d = (a - b).rem_euclid(std::f64::consts::TAU);
    d.min(std::f64::consts::TAU - d)
}

Exercises

The stubs are in rust/ch17-open-problems/src/exercises.rs.

cargo test -p ch17-open-problems --test exercises ex17_1

cargo test -p ch17-open-problems --test exercises ex17_2

cargo test -p ch17-open-problems --test exercises ex17_3

Further reading

• The WAM survey's section on open challenges (Lu et al. 2026, arXiv 2609.16074).
• The April 2026 comprehensive review's challenges and future directions (Psiris et al. 2026, arXiv 2604.15395).
• The robustness study comparing world-action models and VLAs (Zhang et al. 2026, arXiv 2603.22078).
• LIBERO-Plus and BeTTER, for what benchmarks hide (Fei et al. 2025, arXiv 2510.13626; Xu et al. 2026, arXiv 2604.18000).
• The industrial readiness survey, for safety and integration as research problems (Kube et al. 2026, arXiv 2603.06749).