Part II: Generative and Multimodal Architectures

"Learning directly from raw text about images is a promising alternative."

Alec Radford et al., "Learning Transferable Visual Models From Natural Language Supervision"

Part I built the machinery for a model that consumes tokens. This part steps off the main text-model spine to follow a parallel track: generation when the object is not naturally a left-to-right string. Images, audio, video, speech, and embodied observations all force the same uncomfortable issue: the model must make something coherent, but the thing being made may not have a single natural order. The autoregressive language model resumes its lifecycle in Part III; the architectures here are its siblings, not its successors.

Part II follows the architectures that grew around that pressure. It opens in Chapter 12 with denoising and flow matching, where generation is learned as a path from noise to structure rather than as the next token in a sequence. That pressure returns to language in Chapter 13, which asks what non-autoregressive and diffusion language models gain, and what they lose when they give up the cache and the clean serving story of autoregression. Chapter 14 and Chapter 15 then move through speech recognition, speech generation, vision-language fusion, image generation, and video. By Chapter 16 the frame widens to world models, robotics, and embodiment, where the missing ingredient is often not a bigger model but a reliable stream of grounded experience.

The part is not a tour of media formats. It is a study of ordering. Whenever a chapter seems to be about images, audio, or robots, ask what sequence the system invented so the model could train, cache, sample, and be evaluated. That invented order is the architecture.