A Field Map and How to Read This Book

The previous chapter followed one request through the stack. Now that line needs to sit inside the whole terrain: where any topic in the book belongs, which claims are settled and which are still argued over, how this book stands next to the comprehensive titles already on the shelf, and why every later component is read design-first, by asking why it has this shape and not merely what it does. That is the map this chapter supplies.

It is organized around four questions a newcomer to the field actually asks: where each piece goes, which parts can be trusted as fact and which are live argument, what this book adds to the books already available, and how to read without drowning.

One map, eleven parts

Chapter 1 walked one task from raw data to a deployed agent in a single pass. Zooming out from that vertical slice shows how the substantive parts, Part I through Part XI, interlock. The cleanest way to hold them in mind is as a stack: the lifecycle of a capability, with compute and data at the bottom, a governed behavior at the top, and at every layer a set of choices that the layer below constrains and the layer above inherits. Figure 2.2 draws that stack.

Read the solid arrows first; they are the build order. Part IX, the accelerators, the networks, the data infrastructure, and the compute frontier, sits at the bottom: it is drawn below Part I because a cluster's shape is the first constraint on what can be trained at all (Chapter 62, Chapter 65). The same part later returns above the model, where capability measurement and verification decide which results can be accepted (Chapter 71, Chapter 72). Part I is the base-model formation layer: it turns that compute into a model by using a scaling law, an empirical formula that predicts loss from model size, data, and compute, to size the run, then data curation, tokenization, architecture, training at scale, and mid-training bridge phases (Chapter 5, Chapter 6, Chapter 7, Chapter 8, Chapter 9, Chapter 10, Chapter 11). Part II takes that same foundation in a second direction, into diffusion, flow matching, and multimodal generation beyond text (Chapter 12, Chapter 15, Chapter 16). Part III turns the base model into a post-trained assistant: demonstrations, behavior specifications, preference rewards, direct preference objectives, verifiable rewards, safety tuning, and synthetic-data loops (Chapter 17, Chapter 18, Chapter 19, Chapter 20, Chapter 21, Chapter 22, Chapter 23). Part IV turns reasoning into an explicit control system: elicit latent work, structure the search, offload pieces to programs and solvers, select with verifiers, train from checkable rewards, distill reasoning traces, and route test-time compute in production (Chapter 24, Chapter 25, Chapter 26, Chapter 27, Chapter 28, Chapter 29, Chapter 30). Part V is the serving problem: turning a trained model into tokens a user can afford to wait for (Chapter 31, Chapter 32, Chapter 33, Chapter 34, Chapter 35). Part VI composes served models into agents with memory, tools, retrieval, and context (Chapter 38, Chapter 39, Chapter 41, Chapter 43, Chapter 44, Chapter 46). Part VII measures whether any of it works, from public benchmarks through uncertainty, human rubrics, judges, factuality, agent trajectories, and release gates (Chapter 47, Chapter 48, Chapter 49, Chapter 50, Chapter 51, Chapter 52, Chapter 53). Part VIII reaches sideways into every layer to ask whether the behavior can be understood, overseen, and secured (Chapter 54, Chapter 55, Chapter 56). Part X reads the ecosystem around the stack: model openness, tooling standards, compute prices, market structure, adoption, and data rights (Chapter 73, Chapter 75, Chapter 76, Chapter 77, Chapter 78, Chapter 79). Part XI is the capstone: it puts the whole stack to work in practice, from choosing a model to wiring, deploying, reviewing, and learning from it in production (Chapter 81, Chapter 88, Chapter 89, Chapter 91, Chapter 92).

The cross-cutting relationships behind the dashed arrows are the reason to read in order, and they are easy to miss. They do not feed one layer into the next; they reach across the stack. Evaluation feeds back into adaptation: a benchmark or a judge that moves changes what the next round of post-training optimizes for. Safety constrains adaptation and orchestration rather than sitting at the end as a coat of paint. Human oversight turns those constraints into product surfaces: approval gates, escalation paths, correction loops, and trace capture. Economics prices pretraining and serving, and that price reaches up the stack to change design choices that look, locally, like pure modeling decisions. Operating contracts reach back from Part XI into serving, evaluation, data, and safety: SLOs, budget caps, incident records, and tenant boundaries become design constraints, not deployment paperwork. Hold the dashed arrows in mind as you read; the reading-instructions section below shows the sharpest one in full.

Solid ground and swamp

A map is only useful if it marks the solid ground separately from the swamp, and a reader's first defense against the noise of this field is knowing which is which.

Some of this book is settled engineering. The transformer block and its attention mechanism (Vaswani et al. 2017), the autoregressive next-token objective (training the model to predict the next token from the ones before it), attention with a key-value cache (the saved per-token attention state reused as the model decodes), mixed-precision training (running the arithmetic in lower-precision floats to save memory and time), and the basic fact that loss falls as a smooth power law in compute, dropping by a roughly fixed ratio each time the compute budget multiplies (Kaplan et al. 2020), are no longer in dispute; they are the shared substrate every lab builds on. A reader can learn them as one learns the memory model of a language: as the rules of the game, fixed for the duration of play.

Much of the frontier is not settled, and the book marks those places rather than papering over them. The compute-optimal ratio of tokens to parameters is a budget-and-deployment-dependent choice, not a constant (Hoffmann et al. 2022) (Chapter 5). One debate is whether reinforcement learning on verifiable rewards teaches genuinely new reasoning or only sharpens what pretraining already put there; Chapter 28 and Chapter 29 argue it out, where work like DeepSeek-R1 makes the case that pure reinforcement learning can incentivize reasoning behavior (Guo et al. 2025). Sparse autoencoders, a tool for decomposing a model's tangled activations into separate, readable units, raise a second: are they the right primitive for reading a model's internal features, or only a convenient but lossy lens? Chapter 54 leaves that open. A third question runs through Chapter 47, Chapter 48, and Chapter 52: whether the thing actually being measured is a benchmark number or the harness that runs it. The signature of the book is that these live debates get a box, with named positions, instead of a confident sentence that hides the disagreement. The interactive figure below draws that smooth power law: loss falling as a straight line on log-log axes as the compute budget grows, with the exponent set by the slider.

Existing shelves, and the gap between layers

A reader deciding where to spend attention deserves to know what this book is not. It is not a book about the AI infrastructure that predates the generative stack: ranking, recommendation, and tabular prediction appear only where they constrain or teach, and Chapter 4 draws that boundary explicitly. Nor is it trying to become the deepest public reference on scaling laws, distributed training, GPU kernels, or LLM serving. DeepMind and the JAX team made the systems view of scaling explicit in How to Scale Your Model, which walks from rooflines through TPUs, GPUs, training, inference, serving, and profiling (Austin et al. 2025). DeepMind and Google also made the scaling-law line concrete in papers such as Chinchilla (Hoffmann et al. 2022). The Hugging Face Nanotron team's Ultra-Scale Playbook is a dedicated guide to training large language models on GPU clusters (Hugging Face Nanotron Team 2025). Stanford's CS336 teaches language modeling from scratch with implementation-heavy assignments that cover tokenization, architecture, systems optimization, scaling, data, and alignment (Hashimoto and Liang 2025). If the question is how to implement tensor parallelism, debug a training run, profile a TPU program, serve LLaMA efficiently, or optimize an attention kernel, those resources are the right shelf.

The same is true one layer up. Chip Huyen's AI Engineering (Huyen 2025) is the application layer done thoroughly: how to build products on top of foundation models you call rather than train, covering evaluation, prompt and context design, retrieval, and deployment from the perspective of an engineer adapting models to a use case. Sebastian Raschka's Build a Large Language Model (From Scratch) (Raschka 2024) takes the laptop-scale route, implementing a GPT-style model end to end in PyTorch, from tokenization through attention to pretraining and fine-tuning, so a reader understands the machine by constructing a small one. Suhas Pai's Designing Large Language Model Applications (Pai 2025) is a broad tour from the ingredients of a language model through fine-tuning, retrieval, and agents.

The gap this book tries to occupy is therefore not "training and serving, but deeper." That claim would be weak. The gap is what links one layer to the next. Training and serving matter here because they set prices, latencies, memory shapes, data requirements, and trust boundaries that later chapters inherit. Which pretraining recipe is economical follows from the serving engine. A benchmark harness, by what it rewards, shapes the post-training examples someone bothers to collect; a verifier is what turns a reasoning trace into training data, or leaves it on the floor. A security boundary fixes the outer limit of what an agent runtime is ever allowed to do. Further down, silicon packaging, HBM supply, electricity delivery, export controls, and geography settle which systems can exist at all (Chapter 67, Chapter 68). Whether a model-produced claim becomes accepted knowledge rather than a merely plausible output rests on verification capacity (Chapter 72). And once a system exists, SLOs, budgets, incident playbooks, and tenant isolation decide whether it can be operated (Chapter 93).

Read that way, the book's distinctive unit is not an isolated chapter, but a constraint arrow. The lower layers are taught only as deeply as needed to make those arrows real, and the upper layers are judged by whether they preserve capability, efficiency, and trust under those constraints. When a chapter needs the full craft of cluster training or serving implementation, it points outward to the specialized shelf rather than pretending to replace it. What the book keeps for itself is the cross-layer causal account: why a physical, economic, safety, or operational constraint reshapes a technical choice that otherwise looks local.

Reading instructions

The map tells you where things are. This last section explains how the chapters are built, so the later reading requires less repeated orientation.

Every substantial chapter moves through the same arc. A chapter opens with the Problem: the essential constraint the component answers, in terms of latency, cost, memory, scale, safety, or correctness. It then gives the Design: the core idea that answers the problem and the principle underneath it. Evolution traces how the design got here, which earlier approaches came first and why they were superseded, through primary sources. Trade-offs names what is sacrificed for what, and where the known gaps are. Implementation appears only when a minimal sketch, a config, or an operational detail clarifies the design, never to reproduce a file. Each chapter closes with Further reading, first-hand sources first. The arc carries an engineering-to-theory motion: the practice is established first, then the foundations that explain it.

Two callouts carry the book's intellectual signature, and learning to read them is worth the small effort. A What's contested box marks where the field genuinely disagrees, with named positions, as in the box above. A Constraint arrow box marks where a lower layer dictates an upper layer's choice. These arrows are the payoff of reading the stack in order rather than dipping into chapters at random, and the next box shows the canonical one.

Try changing how many tokens each model serves over its life and watch the crossover move: a smaller, deliberately over-trained model costs more to train but less per token, so past some serving volume it wins on total cost.

import numpy as np
import matplotlib.pyplot as plt

# Two models assumed to reach the same quality, compared on lifetime cost.
# Training FLOPs ~ 6*N*D, inference FLOPs ~ 2*N per generated token.
big_N, big_D = 70e9, 1.4e12      # compute-optimal larger model
small_N, small_D = 13e9, 12e12   # smaller model over-trained on more tokens

served = np.logspace(11, 15, 200)  # tokens served over the model's life
big_cost = 6 * big_N * big_D + 2 * big_N * served
small_cost = 6 * small_N * small_D + 2 * small_N * served
cross = served[np.argmin(np.abs(big_cost - small_cost))]

plt.loglog(served, big_cost, label="compute-optimal 70B")
plt.loglog(served, small_cost, label="over-trained 13B")
plt.axvline(cross, ls="--", color="gray")
plt.xlabel("tokens served over life"); plt.ylabel("total FLOPs"); plt.legend()
print(f"crossover near {cross:.2e} served tokens")
plt.show()

Two motifs recur without needing a callout, and noticing them ties distant chapters together. The three loops, training, inference, and agentic, are one control structure seen at three timescales: a weeks-long loop that turns compute into a model, a millisecond loop that turns a prompt into tokens, and a multi-step loop that turns a goal into actions. Figure 2.4 draws all three side by side so the shared shape is visible: each is an input that feeds a step that produces an output that returns to the input, and only the payload and the timescale differ.

The capability, efficiency, trust lens closes many chapters: a technique buys capability, or efficiency, or trust, and usually trades one against another, and naming which it buys keeps the comparison fair.

A reader does not have to read strictly front to back. The parts are written to interlock, so following the solid arrows of the map, base-model formation before adaptation, serving before orchestration, gives the constraint arrows their force and is the recommended path. Reading by interest is fine too; the cross-references will point back to whatever lower layer a chapter depends on. Either way, keep the four questions of this chapter in hand: where it goes on the map, whether it is solid or contested, what this book adds, and which way the arrows run.

Further reading