Beyond Text: Multimodal, World Models, and Embodiment

A generated video, a learned simulator, and a robot policy can all use the same visual backbone. They solve different problems. The video generator predicts a plausible observation. The simulator predicts how a state changes under a chosen action. The policy chooses an action and must survive the result. Treating those three outputs as evidence of one general “world model” obscures the contracts that can actually be tested. Predicting plausible observations is not the same as predicting what changes under an action.

This chapter separates those contracts. It defines the state and dynamics needed for planning, explains why visual realism is weak evidence for physical prediction, and follows the consequences into robot data, action representations, control frequency, and evaluation. The central question is not whether a model accepts pixels. It is whether its predictions remain useful when an agent intervenes and the environment answers back.

Separate generation, dynamics, planning, and control

Four capabilities are often bundled under “embodied intelligence”:

Success in one row does not imply success in the next. A video can look coherent without answering what would have happened under another action. A dynamics model can predict a bounded laboratory task without supporting open-world planning. A planner can exploit errors in its learned model, and a policy can fail because sensing, latency, calibration, or low-level control differs from training.

The word state also needs care. The physical state of a room is not directly available to a camera. A model constructs a task-dependent state from its observation history. That state may omit texture while preserving object pose, or omit geometry while preserving features useful for reward prediction. There is no requirement that one representation serve every objective.

Give a world model an action interface

Neural world models have a longer history than current video generators. Ha and Schmidhuber combined a compressed visual representation with recurrent dynamics and trained policies inside the learned model in bounded game environments (Ha and Schmidhuber 2018). PlaNet later learned stochastic latent dynamics from pixels and used online planning for continuous-control tasks (Hafner et al. 2019). MuZero made the task dependence explicit: its hidden dynamics predicted rewards, values, and policies useful for search without reconstructing the observation (Schrittwieser et al. 2020). These systems differ substantially, but each gives actions a defined role in state transition.

One general deterministic skeleton is

$$z_t=e_\phi(o_{1:t}), \qquad \widehat z_{t+1}=f_\theta(z_t,a_t), \qquad (\widehat r_t,\widehat d_t)=h_\psi(z_t,a_t).$$

Here (t) is a discrete time index; (o_{1:t}) is the observation history through time (t); (z_t) is the learned state; (a_t) is the action applied at time (t); (e_\phi) is an encoder with parameters (\phi); (f_\theta) is a learned transition with parameters (\theta); (\widehat z_{t+1}) is its predicted next state; and (h_\psi), with parameters (\psi), predicts reward (\widehat r_t) and termination probability (\widehat d_t). A decoder (g_\omega(z_t)=\widehat o_t) can render a predicted observation, but planning does not inherently require one. Stochastic world models replace the point prediction with a distribution over possible next states.

A representation-prediction objective can take the form

$$\mathcal{L}_{\mathrm{dyn}} = \sum_{t=1}^{T-1} D\!\left( f_\theta(e_\phi(o_{1:t}),a_t), \operatorname{sg}[e_\phi(o_{1:t+1})] \right).$$

In this example, (T) is the trajectory length, (D(\cdot,\cdot)) measures the discrepancy between two representations, and (\operatorname{sg}) is a stop-gradient operator: its argument supplies a target but receives no gradient through that occurrence. This is a template, not a universal world-model loss. Other systems reconstruct observations, predict distributions, rewards, values, or combinations of them. The invariant is the conditional question posed to the model, especially whether the transition depends on (a_t).

Visual realism is not an intervention test

A video generator commonly models a distribution such as

$$p_\theta(o_{t+1:t+H}\mid o_{1:t},c),$$

where (o_{t+1:t+H}) is a future sequence of (H) observations, (o_{1:t}) is the available visual context, (c) is a text or other condition, and (\theta) denotes model parameters. This objective can reward texture, motion, composition, and common event sequences. It need not identify which hidden dynamics produced the observations, and it supplies no action variable unless the training data and model explicitly provide one.

That distinction is partly statistical. Passive video records what happened under the behavior that generated the dataset. It rarely shows the same state followed by several controlled interventions. Different dynamics can therefore fit the observed clips while disagreeing about an untried action. More video may improve coverage, but coverage alone does not make those counterfactuals identifiable.

Recent evaluations make the limitation concrete without proving an impossibility result. Physics-IQ separates perceptual realism from performance on recorded physical experiments and found little relationship between the two for the video models it tested (Motamed et al. 2026). Controlled synthetic experiments on physical-law extrapolation likewise found strong in-distribution imitation and weak out-of-distribution generalization in the studied generators (Kang et al. 2025). These are benchmark results for particular models and prompts. They justify testing physics separately from appearance; they do not show that scale or video learning can never acquire useful dynamics.

At least four evaluations are needed before calling a generator a world model:

The last two tests are stronger because they expose the model to decisions that can exploit its errors. A plausible rollout is useful for presentation. A decision model must remain accurate along trajectories selected by a planner, including trajectories unlike the passive training distribution.

Choose what the model must preserve

World-model representations fall along a practical spectrum rather than into three mutually exclusive camps:

Observation-space and latent prediction can coexist: a system may plan in a latent state and decode only for training or inspection. Structured geometry can be one component of a learned latent rather than a competing architecture. The right representation is the smallest one that preserves the distinctions needed by the downstream decisions.

V-JEPA 2 illustrates the latent route. Its encoder was pretrained on more than a million hours of image and video data. A separate 300-million-parameter, action-conditioned predictor was then trained on fewer than 62 hours of robot video and used in a model-predictive-control loop for image-goal pick-and-place experiments on Franka arms (Assran et al. 2025). This result demonstrates planning in the reported tabletop setting. It does not establish general manipulation or open-world physics, and the action-conditioned robot stage remains essential. Passive video supplied a representation prior, not a complete controller.

Planning turns prediction errors into decisions

Given a learned state (z_t), a finite-horizon planner can choose

$$a^*_{t:t+H-1} = \arg\max_{a_{t:t+H-1}} \mathbb{E} \left[ \sum_{k=0}^{H-1} \gamma^k\widehat r(z_{t+k},a_{t+k}) \right], \qquad z_{t+k+1}=f_\theta(z_{t+k},a_{t+k}).$$

Here (a^*{t:t+H-1}) is the selected action sequence; (H) is the planning horizon; (k) indexes predicted steps; (\gamma\in[0,1]) discounts later rewards; (\widehat r) is the learned or specified reward; the expectation is over stochastic predictions; and (f\theta) supplies the state transition. Model-predictive control executes only the first action or a short prefix, reads the next real observation, updates (z_t), and plans again.

Long rollouts compound transition error. More subtly, optimization searches for action sequences that look unusually good to the model, so it can find flaws that random held-out trajectories do not expose. Useful evaluation therefore includes planner-selected rollouts, uncertainty under distribution shift, and real closed-loop return, not only one-step prediction loss.

A shared backbone is not yet a world model

Chapter 15 separated input representation, fusion, output objective, and system composition. None of those choices alone creates a world model. A single backbone may process text, images, audio, and video yet have no action-conditioned transition. Conversely, a small robot dynamics model may be a valid world model for one workcell while accepting only proprioception and a depth image.

Native multimodal training can still help embodiment. Language supplies task descriptions and semantic priors; video supplies appearance and motion priors; depth and proprioception constrain geometry and robot state. The scientific claim to test is transfer: after holding robot interaction data and evaluation fixed, does adding a modality improve closed-loop behavior on new objects, instructions, or environments? Merely placing modalities in one sequence does not answer that question.

A robot policy has a physical output contract

A vision-language-action model (vision-language-action model (VLA)) extends a perception-language model with a policy over robot commands. A chunked policy can be written as

$$\pi_\eta(a_{t:t+K-1}\mid o_{1:t},q_t,\ell),$$

where (\pi_\eta) is a policy with parameters (\eta); (o_{1:t}) is the observation history; (q_t) is proprioception such as joint positions and velocities; (\ell) is the language instruction; (K) is the predicted action horizon; and (a_{t:t+K-1}) contains (K) robot actions. The actions may be discrete bins, continuous joint targets, end-effector poses, gripper commands, or a trajectory distribution. Those choices are not interchangeable across robot bodies.

RT-2 demonstrated one discrete route by expressing actions as tokens and co-fine-tuning on web-scale vision-language data and robot trajectories (Brohan et al. 2023). OpenVLA provided an open 7-billion-parameter implementation trained on 970,000 real-world robot demonstrations (Kim et al. 2024). These models can reuse semantic representations from web data, but contact, latency, calibration, and action semantics still come from embodied data and the control stack.

Continuous policies provide another route. Diffusion Policy models a multimodal distribution over continuous action sequences and replans in a receding-horizon loop (Chi et al. 2023). Action Chunking with Transformers (ACT) predicts (K) future actions at once and combines overlapping predictions with a temporal ensemble (Zhao et al. 2023). Chunking shortens the policy's effective prediction horizon, but executing a longer prefix without new observations also reduces feedback. Here, (K) denotes the predicted chunk length; it remains separate from the replanning interval and low-level control rate.

Count control ticks and model calls

For an episode lasting (D) seconds at a low-level control rate of (F) hertz, the number of control targets is

$$N_{\mathrm{ctrl}}=\lceil FD\rceil.$$

Here (N_{\mathrm{ctrl}}) is the number of control ticks, (F) is the number of commands executed per second, and (D) is episode duration. A 60-second episode contains 600 targets at 10 hertz and 3,000 at 50 hertz. These are correlated control targets, not 600 or 3,000 independent demonstrations.

If a large policy runs once every (R) control ticks and predicts a horizon of (K) actions, then

$$N_{\mathrm{model}} = \left\lceil\frac{N_{\mathrm{ctrl}}}{R}\right\rceil, \qquad N_{\mathrm{pred}}=K N_{\mathrm{model}}.$$

Here, (N_{\mathrm{model}}) counts large-model calls, (R\ge1) is the replanning interval in control ticks, and (N_{\mathrm{pred}}) counts all action positions predicted, including overlapping positions that may be replaced before execution. The runnable makes that distinction explicit for a hypothetical 10-second controller.

from math import ceil

duration_s = 10
control_hz = 50
replan_every = 4
prediction_horizon = 16

control_ticks = ceil(duration_s * control_hz)
model_calls = ceil(control_ticks / replan_every)
predicted_positions = model_calls * prediction_horizon

print(f"executed control ticks: {control_ticks}")
print(f"large-model calls:      {model_calls}")
print(f"predicted positions:    {predicted_positions}")
print(f"model-call reduction:   {control_ticks / model_calls:.1f}x")

This accounting does not promise a fourfold latency improvement. A low-level controller must still issue 500 commands, overlapping predictions add work, and sensing plus actuation have their own deadlines. It shows why robotics systems often separate a slower semantic policy from a faster controller rather than forcing one model to own every timescale.

Robot data is heterogeneous before it is scarce

There is no sound conversion from language tokens to robot trajectories. A trajectory contains many temporally correlated sensor readings and commands; its information depends on task diversity, embodiment, reset procedure, operator, camera placement, and sampling rate. Claims that robotics has a fixed fraction of “the data in language” hide those units.

The available datasets still show a scale and diversity problem. Open X-Embodiment standardized data from 22 robot types, 21 institutions, and 527 skills so cross-robot training could be studied (Open X-Embodiment Collaboration 2023). DROID collected 76,000 demonstrations, about 350 hours, across 564 scenes with a common Franka manipulation setup (Khazatsky et al. 2024). OpenVLA's 970,000 demonstrations are large by robot-learning standards and tiny beside web corpora only in a loose sense. Size is also the wrong axis: each robot dataset samples a narrow set of bodies, controllers, tasks, and environments.

Five data sources trade one missing variable for another:

Simulation-to-real methods can randomize textures, geometry, dynamics, and sensors to reduce reliance on one simulator configuration (Tobin et al. 2017). They do not eliminate the need for real trials. Passive video can pretrain perception or motion representations, as V-JEPA 2 shows, but converting a human hand motion into safe robot torques is a separate learning and calibration problem. Synthetic rollouts are most useful when real data validates where the model is trustworthy.

The useful flywheel is therefore not “generate unlimited robot data.” It is a measured loop: collect a failure, reproduce or approximate it, update the model, and retest on real hardware. Dataset lineage matters because a success rate can rise after near-duplicate scenes or operator habits leak across the split.

Evaluate the closed loop

Offline action prediction is a development metric, not a deployment result. A small per-step error can move the policy into states absent from the demonstrations, after which later predictions are evaluated on inputs the policy helped create. Closed-loop trials expose that feedback.

An embodied evaluation should report at least:

Every rate needs a denominator and trial protocol. “90% success” can mean nine of ten hand-picked attempts or 900 of 1,000 preregistered trials across several sites. Report the number of trials, task distribution, reset policy, operator involvement, hardware version, and uncertainty. For a generalist policy, a mean across tasks should not hide tasks with zero success or unacceptable safety events.

World-model evaluation should follow the same hierarchy. One-step latent error tests the learned transition locally. Open-loop rollouts test compounding error. Counterfactual action tests examine the intervention interface. Planning return tests whether optimization can use the model. Real closed-loop trials test the whole system. No single perceptual metric substitutes for that ladder.

The evidence boundary

Moving beyond text does not reveal one missing ingredient called grounding. It reveals a stack of contracts: observations must preserve task-relevant state, dynamics must respond correctly to actions, planning must tolerate model error, and control must meet the timing and safety requirements of a particular body. Video, language, simulation, and robot experience can contribute to that stack, but their data are not interchangeable.

The durable claim is narrower than “a video generator becomes a world model at scale.” Learned representations and dynamics already support planning in bounded domains, and web video can reduce the robot data needed for some experiments. The open problem is reliable transfer across environments, tasks, bodies, and interventions. Progress should be measured at those boundaries, with closed-loop trials anchored to real hardware.

Further reading