Deployment, evaluation and industrial readiness

Why this chapter exists

A model that works in a paper is not yet a model that works in a factory. Between the two lie three kinds of engineering. The model must run fast enough, on hardware the robot can carry or reach. Someone must measure whether it works, in a way that predicts how it will behave on the job. And it must fit into a safety case that a regulator, an insurer and a works council will accept. This chapter covers each in turn, at the level of what a deployment team needs to ask, not how to implement the answers.

After this chapter you will be able to:

• break a robot's reaction time into its parts with figure 15.1, and explain why long action chunks executed synchronously are dangerous;
• choose where to run each part of a model, on the robot or in the cloud, and say what each choice depends on;
• say what each common benchmark measures and does not, and read an industrial readiness assessment.

The latency budget

Every pass of a VLA has a cost in time: capture an image, prepare it, run the vision-language backbone, run the action head, send the result to the motor controller. Figure 15.1 uses illustrative timings, about 130 milliseconds on the robot, but the structure is general. A cross-accelerator study from April 2026 found the same two phases in real VLAs: a compute-bound backbone followed by a memory-bound action expert, which leaves hardware underused. It also found that right-sized edge devices can meet control rates at lower cost and energy than flagship GPUs (Zhou et al. 2026, arXiv 2604.24447).

The number that matters for safety is not one pass, but how long the robot is blind to a change. With action chunking (chapter 4), a robot that stops to think executes a whole chunk before it looks again. If the chunk lasts two seconds, a person who steps into the workspace just after the robot looked is not seen for two seconds, plus the time to think about it. Asynchronous execution looks again as soon as the last pass finishes, so the worst case is about two passes, whatever the chunk length. Real-time chunking does this for flow-based VLAs without retraining (Black et al. 2025, arXiv 2506.07339), and LingBot-VA uses asynchronous inference for the same reason (Li et al. 2026, arXiv 2601.21998). The other tools that shrink the budget are fewer denoising steps, quantisation, and export to optimised runtimes. GR00T N1.7 adds full-pipeline ONNX and TensorRT export for desktop GPUs and edge platforms (NVIDIA 2026). One industry survey reports quantised VLAs running at 10 to 25 Hz on consumer-grade GPUsindustry survey (SVRC (Robotics Center of Silicon Valley) 2026).

Where to run the model

There are three options. All on the robot makes timing predictable and removes the network as a failure mode. Google DeepMind's case for Gemini Robotics On-Device is exactly that it is "helpful for latency sensitive applications" and robust "in environments with intermittent or zero connectivity" (Google DeepMind 2025). NVIDIA's GR00T platform deploys post-trained policies to Jetson Thor for on-device inference (NVIDIA Technical Blog 2026). All in the cloud buys the biggest models and fastest GPUs, and makes every motion depend on the network. The split keeps the controller on the robot and puts the reasoning in the cloud. Gemini Robotics hosts its backbone in the cloud with a local action decoder on the robot (Gemini Robotics Team et al. 2025, arXiv 2503.20020), and the embodied reasoning models of chapter 10 sit above an on-robot controller. In the split, a network stall delays a plan, not a motion, which is usually the right failure to have.

Evaluation

Chapter 11 showed how many trials it takes to tell two policies apart. This section is about what the trials should be. Figure 15.3 places the common benchmarks by two properties: simulated or real, and short or long tasks.

Two findings sharpen the picture. LIBERO-Plus perturbed the setups VLAs are usually tested on, changing camera viewpoints, initial robot states, lighting and more. Performance dropped "from 95% to below 30% under modest perturbations", and the models largely ignored their language instructions (Fei et al. 2025, arXiv 2510.13626). A study of the BEHAVIOR-1K household challenge argued that scoring only the final state of objects "can potentially exaggerate reported performance", and proposed counting safety violations along the way (Rasouli et al. 2026, arXiv 2604.21192). On the real-robot side, RoboArena crowd-sources double-blind pairwise comparisons across evaluators who choose their own tasks, which scales diversity (Atreya et al. 2025, arXiv 2506.18123). TRI's blind A/B trials show what rigour looks like inside one lab (TRI LBM Team et al. 2025, arXiv 2507.05331). None of these measures contact quality, or recovery over a long task on a real robot.

Industrial readiness

A March 2026 survey asked a different question: not how well models do on benchmarks, but how well their papers address what industry needs. It distilled the industrial literature into eleven implications, from adaptability and safety to real-time performance and data, and operationalised them as 149 criteria. It then assessed 324 manipulation-capable models through 48,276 criterion-level decisions, made by an LLM-assisted pipeline validated against expert judgements (Kube et al. 2026, arXiv 2603.06749). Figure 15.4 shows its five highest-rated models.

The highest-rated model, Gemini Robotics 1.5, fulfils 12 percent of the criteria overall, with 44 percent on adaptability and flexibility and none on precision, real-time performance or cost-effective integration (Kube et al. 2026, arXiv 2603.06749). The survey's reading is that recent work advances "one or two enabling dimensions at a time". Gemini's lead comes from combining a reasoning model with a VLA, which suggests that near-term deployments will be layered systems (Kube et al. 2026, arXiv 2603.06749). Its scores rate papers, not deployed systems, and a vendor may have solved in practice what it did not publish, so the numbers are a lower bound on readiness and an upper bound on public evidence.

Safety and certification

Industrial robots operate under functional-safety standards. The survey notes the role of ISO 10218 for the safe installation and operation of robots, and ISO/TS 15066 for collaborative operation and contact (Kube et al. 2026, arXiv 2603.06749). These standards were written for systems whose behaviour can be specified, analysed and tested part by part. A modular pipeline fits that pattern: perception, planning and control each have a specification, and much of the chain can sit inside the safety case. A learned end-to-end policy does not. It has no specification beyond its training data, and its failure modes cannot be enumerated in advance.

The practical answer today is the one in figure 15.5. Keep a conventional, certifiable safety layer, such as speed and separation monitoring and force limits, between the learned policy and the motors, independent of it and able to stop it. One of the survey's criteria describes exactly this: a system supervised "by independently certified controllers/modules" that can reject unsafe commands (Kube et al. 2026, arXiv 2603.06749). Model makers say the same in their own terms. Google's model card for Gemini Robotics ER 2 asks users not to use the robotics models for safety-critical applications such as healthcare and transportation (Google DeepMind 2026). This is the modular pipeline's advantage that learned policies gave up, and chapter 17 returns to it as an open problem.

In code

The crate's budget is a list of stages and a sum. The value is in making each stage's time explicit, so it can be measured and argued about.

/// One stage of the pipeline from camera to motor, with its time in milliseconds.
#[derive(Clone, Copy, Debug)]
pub struct Stage {
    pub name: &'static str,
    pub ms: f64,
}

/// An illustrative on-robot pipeline for a VLA with a flow-matching action expert.
pub fn on_device() -> Vec<Stage> {
    vec![
        Stage { name: "camera frame", ms: 33.0 },
        Stage { name: "preprocess and transfer", ms: 8.0 },
        Stage { name: "VLM backbone", ms: 60.0 },
        Stage { name: "action expert, 10 steps", ms: 30.0 },
        Stage { name: "send to controller", ms: 2.0 },
    ]
}

/// The same pipeline with the model in a data centre: a faster GPU, plus the network both ways.
pub fn in_the_cloud(round_trip_ms: f64) -> Vec<Stage> {
    vec![
        Stage { name: "camera frame", ms: 33.0 },
        Stage { name: "preprocess and transfer", ms: 8.0 },
        Stage { name: "network round trip", ms: round_trip_ms },
        Stage { name: "VLM backbone", ms: 25.0 },
        Stage { name: "action expert, 10 steps", ms: 12.0 },
        Stage { name: "send to controller", ms: 2.0 },
    ]
}

/// Total time from a frame being captured to its actions reaching the controller.
pub fn latency_ms(stages: &[Stage]) -> f64 {
    stages.iter().map(|s| s.ms).sum()
}

/// Round-trip times in milliseconds for a run of control cycles over a shared network:
/// mostly near `base`, with occasional long stalls. Deterministic, so runs can be compared.
pub fn network_samples(base: f64, n: usize) -> Vec<f64> {
    (0..n)
        .map(|i| {
            let x = ((i as u64).wrapping_mul(0x9E37_79B9_7F4A_7C15) >> 40) as f64 / (1u64 << 24) as f64;
            if x > 0.97 { base * 8.0 } else if x > 0.85 { base * 2.5 } else { base * (0.8 + 0.4 * x) }
        })
        .collect()
}

Exercises

The stubs are in rust/ch15-deployment/src/exercises.rs. This chapter is equation-free; the exercises are arithmetic about time and coverage.

cargo test -p ch15-deployment --test exercises ex15_1

cargo test -p ch15-deployment --test exercises ex15_2

cargo test -p ch15-deployment --test exercises ex15_3

Further reading

• The industrial readiness survey, the best checklist for a deployment review (Kube et al. 2026, arXiv 2603.06749).
• LIBERO-Plus, for what perturbations reveal about benchmark scores (Fei et al. 2025, arXiv 2510.13626).
• RoboArena, for distributed real-world evaluation (Atreya et al. 2025, arXiv 2506.18123).
• How VLAs (really) work in open-world environments, for safety-aware metrics (Rasouli et al. 2026, arXiv 2604.21192).
• Real-time chunking, for asynchronous execution without retraining (Black et al. 2025, arXiv 2506.07339).
• The cross-accelerator study, for choosing deployment hardware (Zhou et al. 2026, arXiv 2604.24447).