Part II · Chapter 16

Beyond Text: Multimodal, World Models, and Embodiment

AuthorChangkun Ou

Reading time~14 min

Text has a data wall: the supply of high-quality tokens is finite, a limit Chapter 70 takes up in full. Leaving text does not remove that wall; it makes the missing data more physical. Video generators can look convincing without learning transferable physics, world-model papers disagree on what the model is supposed to represent, and robotics has no internet-scale action corpus to inherit. Generation, perception, and control therefore break at the same place: the absence of a grounded data flywheel and a transferable internal model of the world. Multimodality, world models, and robotics test whether the foundation-model recipe is general, or whether it worked because text and images arrived with free data attached. The early evidence leans toward the latter: the recipe was carried more by free data than the field liked to admit.

Figure 16.1. Schematic of why leaving text changes the learning problem: action horizons grow while dense human supervision becomes scarcer. Idealized placement of modalities, not measured data.

Looking real and being right are different things

The most striking artifacts of the period are the video generators. They produce seconds of photorealistic, physically plausible motion from a sentence. How those generators are built, diffusion transformers over spacetime latent patches, is the subject of Chapter 15; the question here is what they understand. The temptation is to read that fluency as understanding, as if a model that can render a ball bouncing has learned that balls bounce. Controlled study says it has not. When a video model is tested out of distribution, on inputs unlike anything it saw in training (here, a physical situation unlike its training trajectories), it fails, and it fails in a revealing way: it imitates the nearest scene it saw rather than applying a rule, and scaling the model and the data does not fix it (Kang et al. 2025). A benchmark built to decorrelate looking-real from being-right confirmed the point from the other side: across the leading generators, physical-understanding scores are severely limited and essentially unrelated to visual realism (Motamed et al. 2025). The realism and the physics are independent axes, and scaling buys the first without the second.

This is the binding constraint of the whole chapter, and it is epistemic rather than engineering. The engine that bought reasoning in language, more data and more parameters, has not been shown to buy a transferable physics prior in pixels. A frontier video generator is a sophisticated predictor of the next frame, and prediction of appearance is not possession of a model of the world.

Constraint arrow

The absence of a learned internal representation of state and dynamics dictates that physics cannot generalize out of distribution: the model falls back to case-based imitation, so adding data and parameters does not buy planning. This is the same shape of limit as the hallucination floor of Chapter 70, one layer over: the objective rewards matching the training distribution, not understanding it, so the residual failure is structural (cross-ref Chapter 5, Chapter 28).

Three answers to "what is a world model?"

If predicting pixels is not enough, what is? The field has split into three architectural answers, and the split is the chapter's organizing fault line.

The first answer is to generate the world anyway, and make the generation interactive and controllable. Real-time world simulators produce navigable scenes a user can act in, and their builders frame them explicitly as a path toward general capability, an unlimited supply of training environments (Google DeepMind 2025). The real limit is coherence: such a simulator holds only about a minute of visual memory before it forgets what it rendered, which is why this line keeps bolting on external memory rather than growing it internally. The second answer abandons pixels. A latent world model predicts in an abstract representation rather than rendering every pixel, on the argument that most pixels are unpredictable detail that wastes the model's capacity, and that planning should happen in the latent space; one such model, pretrained on a million hours of video and post-trained on under a hundred hours of robot data, drove a real arm zero-shot (with no task-specific training) (Assran et al. 2025). The third answer, newer, reconstructs the world as explicit persistent three-dimensional geometry, so that consistency is a property of the representation rather than something the model must remember. Where the first answer patches the coherence problem and the second sidesteps rendering, the third argues that persistent geometry is the world model.

Figure 16.2. Three architectural answers to what a world model is, each a different response to the out-of-distribution physics wall. The generative path renders and patches coherence with external memory; the latent path predicts in abstraction and plans there; the explicit-3D path makes persistent geometry the representation.

It helps to remember that latent world models for planning are not new: an earlier model-based reinforcement-learning lineage already learned a latent world model (a learned internal simulator of the environment) and planned inside it to solve bounded control tasks. That history sharpens the critique rather than dismissing it: latent world models plainly can plan in closed domains; the open question is whether any of the three paths scales to open-world physics, which none has yet shown.

The cautionary tale is what happened when generation shipped as a consumer product. A flagship video app, technically dazzling, was wound down in 2026 after reportedly burning millions of dollars a day in compute, by one analyst estimate over ten million (Liu 2025), and colliding with copyright, even as the underlying model survived elsewhere. It is the chapter's thesis made vivid: product-grade realism can be fully decoupled from both sustainable economics and world-model value.

Native multimodality, and how to fuse it

A parallel frontier asks not whether a model understands the world but whether one model can perceive and produce across modalities at all, rather than bolting a vision encoder and an image generator onto a frozen language model. What changed in this period is that early fusion, training a single backbone jointly on text and pixels from the start, became mainstream for the general-purpose flagships, displacing the adapter-bridged stack. The live disagreement is the representation of vision inside that backbone. One camp recasts images as discrete tokens so the whole model is a clean next-token predictor, accepting the fidelity lost to quantization, and a notable model in this camp trained end-to-end on video frames as a "world learner" that predicts the next state across vision and language. The other camp keeps vision continuous and generates pixels through flow-matching to avoid the tokenizer's quality cliff, at the cost of a more complex architecture. Whether deep joint training actually buys cross-modal transfer, or merely co-locates two capabilities that remain internally separate, is the unsettled scientific question underneath the engineering.

Robotics has no internet

The harshest form of the data wall is in robotics. Language and vision were handed a corpus for free; action was not. Every robot trajectory must be manufactured, and the gap to text is not a factor of two but roughly five orders of magnitude: the largest shared cross-embodiment dataset is estimated at around one two-hundred-thousandth of the token volume of a language-model corpus (Authors 2025; Open X-Embodiment Collaboration 2023). The consequence shows up directly in how the leading models are trained. A vision-language-action model (VLA) that generalized across homes drew the overwhelming majority of its training examples, by one accounting more than 97 percent, from data that was not mobile-manipulation robot data at all, borrowing from static arms, cross-embodiment logs, and web vision-language pairs (Physical Intelligence 2025). The architecture is shaped by scarcity: it is built around what borrowed priors can supply because the native data does not exist.

The lineage of these models is the transplant of the language recipe onto control. It began by fine-tuning a vision-language model on robot trajectories encoded as tokens, on the founding claim that web knowledge transfers to control (Brohan et al. 2023), continued through an open, reproducible baseline (Kim et al. 2024), and reached models that transfer one policy across heterogeneous robots by sharing a motion representation (Google DeepMind 2025). Whether the web priors genuinely transfer to physical control, or only supply object and instruction semantics while the scarce robot data does the real work of contact dynamics and closed-loop control, is the field's central and unresolved transfer debate.

Constraint arrow

No internet-scale action corpus dictates the design of the whole stack. Co-training mixtures are dominated by non-robot data, cross-embodiment pooling becomes architecturally necessary to fake scale, and the closed-loop control rate (how often the robot senses and issues a fresh command), on the order of 50 to 100 Hz, forces the model to split into a slow reasoner and a fast on-robot executor with action chunking (predicting a short burst of actions at once rather than one step at a time) and asynchronous inference, which ties the constraint to concrete on-robot silicon (cross-ref Chapter 33). Data scarcity at the bottom reshapes architecture, serving, and hardware all the way up.

Four ways to manufacture a flywheel

Because the data does not exist, the frontier of robotics is really a contest of strategies to manufacture it, and every direction is one such strategy rather than an independent advance.

The first is to collect real data at scale through teleoperation, and one effort pushed to hundreds of thousands of hours, growing tens of thousands of hours per week (Generalist AI 2025). Its ceiling is human real time: a person drives one robot for one hour to make one hour of data, so the route scales linearly and cannot reach internet scale. A second strategy manufactures data in simulation and by video augmentation, expanding scarce real hours by an order of magnitude; that closes the data gap on paper but reopens the reality gap as the new binding constraint. Mining passive human footage is a third escape, learning from internet video and learned world models to sidestep teleoperation altogether. The fourth route, and the one the leading labs increasingly bet on, is wearable human-activity capture: pretrain on cheap egocentric recordings of people doing tasks, with no robot data in pretraining at all, then post-train on an hour of robot data per task. This last approach directly attacks the human-real-time ceiling of teleoperation, and its emergence in 2026 is a reminder of how fast this frontier moves.

Figure 16.3. Four rival strategies to manufacture the action-data flywheel that robotics never got for free. Each accepts the same bottleneck and bets on a different escape; each reopens a different downstream problem (the reality gap, the transfer question, human real time).

A claimed empirical scaling law for robotics, downstream error falling as a power of pretraining data, would reframe the whole field from an algorithms problem into a data-manufacturing problem, mirroring the Part I narrative. The accurate reading is more guarded than the headline: the reported exponents vary by task rather than collapsing to one clean universal value, the result comes from a single organization on proprietary data without independent replication, and power laws in error are easy to fit and hard to extrapolate to open-world reliability. Alongside the data race, real-world reinforcement learning returned: letting a robot learn from its own successes and failures, rather than only from demonstrations, was shown to roughly double throughput and cut failures on hard long-horizon tasks, which partly answers the generalist-versus-specialist debate in favor of on-task adaptation.

Nobody can grade it yet

The deepest shared problem is measurement, and it is distinct from the benchmark debates of Chapter 47 and Chapter 52. For generative and world models there is no agreed metric for long-horizon consistency, controllability, or physical fidelity: perceptual-quality scores rank the products people enjoy, while physics benchmarks deliberately decorrelate from looking good, and the two do not agree. For embodied policies the situation is worse, because real physical trials are the only fully trusted signal and they do not scale across the task diversity a generalist targets, while simulated and real-to-sim evaluations scale but carry a domain gap that makes them an unreliable ranking signal. Until there is a metric that can tell a better simulator from a better world model, and a better demo from a more reliable policy, "world models as a path to physical reasoning" remains a bet rather than a result.

What's contested

Does scaling video generation yield a world model or a better pixel predictor? The emergence camp reads scale as buying physics; the controlled studies read it as case-based imitation that fails out of distribution. The evidence favors the skeptics.
Generative, latent, or explicit-3D? Render the world, represent it in latent space, or reconstruct it as persistent geometry. The split is about whether a world model should render, merely represent, or be a geometry, and none has shown open-world planning.
Do web priors transfer to control? One camp credits emergent semantic generalization to web pretraining; the other says the priors supply semantics but not contact dynamics, and the scarce robot data does the real work.
Collect, simulate, mine video, or capture humans? Four manufacturing routes for the missing action flywheel, each accepting the same bottleneck and reopening a different downstream gap.
Are robotics scaling laws real? A single-organization power law versus the objection that it is unreplicated and hard to extrapolate to reliability.
Can generalist policies be measured at all? Real trials do not scale; simulated proxies carry a domain gap. The evaluation crisis is itself the open problem.

The missing data flywheel

Here the foundation-model recipe meets the limit that text did not expose. Capability is bounded by grounding: a model can render or describe the world far better than it can predict or act in it, because the data that grounds prediction and action in physics was never free. The efficiency question is what that grounding costs to manufacture, in teleoperation hours, simulation compute, or wearable capture, and whichever route produces it cheapest may decide the field. Trust is the part still unmeasured: we cannot yet certify that a world model understands or that a policy is reliable, so deployment outruns evaluation. Across generation, perception, and control, the same wall stands, the absence of a grounded data flywheel and a transferable internal world, and the field is best understood as a set of rival bets on how to get around it. The final chapter turns from these specific frontiers to the instrument that is supposed to tell us whether any of them, or the whole enterprise, is actually working.