The Whole Stack in One Pass

AI systems have two histories. The first happens before a user arrives: data is collected, a model is trained, its behavior is refined, and its weights are deployed. The second begins when a request reaches the deployed system: a model service generates tokens, an agent runtime invokes tools, and tests or other observations determine what happens next. These histories meet at deployment, but they are not the same process.

This chapter follows both. The distinction matters because the expected runtime workload can change earlier design choices. A model that is cheap to train may be expensive to serve. An agent that can call tools needs stronger evaluation and security boundaries than a model that only produces text. Later chapters develop each mechanism in detail; here the aim is to establish the connections.

The running task is simple: a user asks an assistant to find a bug in a small repository, propose a fix, and explain it. Building that capability requires training. Executing the request requires serving, tools, and feedback from the repository. Keeping those phases separate will make their dependencies easier to see.

Two released systems provide concrete architecture examples. Llama 3 uses a dense Transformer, which applies nearly all of the model's layers and parameters to every token (Grattafiori and others 2024). DeepSeek-V3 is a mixture-of-experts (MoE) model that activates only a few expert sub-networks for each token: 37 billion of its 671 billion parameters are active for that token (DeepSeek-AI 2024). These models are not a controlled comparison of equal capability. They are useful because they expose two different cost structures. Later releases continued the move toward sparse capacity: Meta introduced its first mixture-of-experts Llama models in April 2025 (Meta AI 2025), and DeepSeek released a preview of its sparse V4 family in April 2026 (DeepSeek-AI 2026).

Building the capability: data and tokens

The user's request does not pass through a training corpus. The capability to handle it was built from one. A training corpus combines text, code, and other data, then removes low-quality records and repeated material. Near-duplicate removal matters because copying the same repository many times would give it more influence than intended. Decontamination checks whether evaluation problems, or close copies of them, have entered the training set. Without that check, a benchmark may measure recall rather than generalization. These choices are the subject of Chapter 6.

Llama 3's largest model was pre-trained on 15.6 trillion tokens (Grattafiori and others 2024), and DeepSeek-V3 on 14.8 trillion (DeepSeek-AI 2024). A token is an integer identifier for a piece of the input, not necessarily a word. The tokenizer and its vocabulary determine whether a variable such as readUserConfig becomes one familiar unit or several smaller pieces. That representation affects sequence length, multilingual text, and how efficiently the model can learn code. Chapter 7 develops this step from bytes to token identifiers.

Pre-training produces a base model

Pre-training repeatedly gives the model a prefix and asks it to predict the next token. Given if err != nil {, for example, a model that assigns high probability to return incurs less loss than one that assigns the probability elsewhere. A training step computes this prediction error, propagates it backward, and updates the model's parameters. Repeating the process over a large corpus produces a base model: a general predictor of likely continuations, not yet a reliable assistant.

For a fixed compute budget, model size and training-token count cannot both grow without limit. Empirical scaling laws estimate how loss changes with parameters, data, and compute (Kaplan et al. 2020). Hoffmann et al. then studied the balance between parameters and tokens that minimizes loss under a training-compute budget (Hoffmann et al. 2022). Chapter 5 explains what these fits can and cannot predict. The Transformer components that perform the prediction, including attention, residual connections, and normalization, begin with Vaswani et al. (Vaswani et al. 2017) and are developed in Chapter 8.

Architecture changes the cost of this process. Llama 3's 405-billion-parameter model is dense, so most parameterized computation runs for every token (Grattafiori and others 2024). DeepSeek-V3 stores a larger pool of expert parameters but activates only part of it for each token. DeepSeek-V3 combines this routing with multi-head latent attention (MLA), an attention design that shrinks the inference-time cache (DeepSeek-AI 2024). Sparse activation reduces arithmetic per token, but the full expert pool must remain available across the serving system's accelerator memory, and routing tokens among experts adds communication and load-balancing work. Chapter 9 examines that trade-off.

At this scale, the training system is part of the model design. Thousands of accelerators must exchange gradients, recover from hardware faults, and avoid waiting on slow workers. Chapter 10 develops these problems using, among other examples, DeepSeek-V3's FP8 mixed-precision training and its reported 2.788 million H800 GPU-hours for the full training pipeline (DeepSeek-AI 2024).

Many training runs add mid-training before post-training. This phase shifts the data mixture toward high-quality or specialized material, such as code and mathematics, or extends the sequence length. For the running example, a code-heavy mixture can improve repository understanding before instruction tuning begins. Chapter 11 explains why this phase is distinct from both broad pre-training and later behavior shaping.

Part I calls this full sequence base-model formation: data preparation, pre-training, and the mid-training bridge that produces the raw model later adapted for use.

Post-training shapes behavior

A base model continues text. A deployed assistant must instead interpret a request, follow constraints, and present a useful result. Supervised fine-tuning (Chapter 17) begins this change by training on demonstrations of desired responses. A behavior specification and preference dataset then state which responses annotators or automated graders prefer (Chapter 18).

Preference training follows two common routes. In reinforcement learning from human feedback, a reward model learns to score responses, and the language model is optimized as a policy to receive higher scores (Chapter 19). Direct preference methods train from preferred and rejected response pairs without a separate reward model (Chapter 20). Safety tuning adds rules about which instructions take priority and when the model should refuse (Chapter 22).

For the bug-fixing task, post-training turns plausible code completion into a sequence of useful behaviors: inspect the repository, state a plan, use tools, and explain a patch. Reasoning-oriented training and test-time methods may also allocate more computation to difficult steps. Part IV follows that path from elicitation and search to verification, distillation, and inference-time routing (Chapter 24 through Chapter 30).

Deployment turns weights into a service

A trained checkpoint is a set of weights. A service must load those weights, accept concurrent requests, schedule accelerator work, and return tokens within a latency budget. Processing the input prompt, called prefill, often demands substantial arithmetic. Generating later tokens, called decode, repeatedly reads model weights and is often limited by memory bandwidth.

During generation, attention needs keys and values computed for earlier tokens. The key-value cache stores this state so the model does not recompute the entire prefix at every step, but the cache grows with the number of tokens and active requests. Chapter 31 introduces this cost; Chapter 32 explains batching and paging the cache. Speculative decoding proposes several tokens cheaply and verifies them with the target model (Chapter 33). Quantization stores selected weights or cache values at lower precision, reducing memory and bandwidth at some risk to accuracy (Chapter 34).

Runtime: the request enters the system

Only now does the user's bug report enter the system. An agent runtime sends the request to the model service, interprets the response as text or a tool call, executes permitted actions, and returns observations to the model. A typical trace reads a file, runs a test, inspects the failure, edits the code, and runs the test again. The model proposes actions; the runtime enforces permissions and executes them; the repository and test runner provide evidence about the result.

Chapter 38 develops this control loop, and Chapter 41 covers the runtime around it. Some systems use retrieval-augmented generation (Chapter 44) to select relevant material from an index. A code agent may instead use file search and repository tools directly. In both cases, Chapter 46 addresses the same finite-context problem: which instructions, files, tool results, and summaries should accompany the next model call.

Figure 1.2 keeps the lifecycle and runtime paths separate. The deployed weights connect them: training produces the artifact that serving loads, while the request itself moves only through the runtime path.

Three processes at different timescales

Three repeated processes recur throughout the book, but only two are nested. Training happens upstream and produces the weights. A model call then decodes one token at a time from those fixed weights. An agent task may contain several model calls, interleaved with tool actions and observations.

The training loop processes batches, computes a loss, and updates parameters with gradients. Its cost is amortized over a released model version. The decoding loop runs inside each model call: every generated token becomes part of the input to the next step, while the weights remain unchanged. Its cost depends on input and output length. The agent loop runs at the task level, where model calls alternate with actions in an environment. Its total cost and failure probability accumulate across the trajectory.

Figure 1.3 shows these relationships. The agent loop contains model calls, and each call contains token-by-token decoding. Training sits to the left as their dependency rather than inside them. This gives three useful accounting units: per model version, per token and model call, and per task.

The processes are separated in time, but their costs are coupled. Expected runtime demand can therefore change a training-time decision.

The lifecycle comparison can be written without pretending that two released models form a controlled experiment. For a design $j$, let

$$C_j(V) = T_j + V I_j,$$

where $C_j(V)$ is total lifecycle cost at serving volume $V$, $T_j$ is the one-time training cost for a model version, and $I_j$ is the incremental serving cost per unit of volume. Suppose design B costs more to train than design A, $T_B>T_A$, but less to serve, $I_B<I_A$. B becomes cheaper only above the break-even volume

$$V^* = \frac{T_B-T_A}{I_A-I_B}.$$

This small model omits hardware utilization, prompt processing, memory capacity, and engineering labor. Its purpose is narrower: it shows exactly when an expected runtime bill can justify a larger up-front training bill.

import numpy as np
import matplotlib.pyplot as plt

# Two hypothetical designs that reach the same target quality.
# Costs are normalized compute units; serving volume is billions of tokens.
designs = {
    "A: cheaper training": dict(train=100, serve=0.80),
    "B: cheaper serving":  dict(train=140, serve=0.35),
}
served = np.logspace(-1, 3, 200)
for name, d in designs.items():
    plt.plot(served, d["train"] + served*d["serve"], label=name)

a, b = designs.values()
break_even = (b["train"] - a["train"]) / (a["serve"] - b["serve"])
plt.axvline(break_even, color="grey", linestyle="--")
print(f"break-even volume: {break_even:.1f} billion tokens")
plt.xscale("log")
plt.xlabel("lifetime serving volume, billions of tokens")
plt.ylabel("normalized lifecycle compute cost")
plt.legend(); plt.tight_layout(); plt.show()

Evaluating the system: capability, efficiency, trust

The three processes describe how the system runs. A second motif provides three separate questions for judging a design. They are not coordinates in a fixed budget, and an improvement in one does not necessarily reduce another.

A passing test supplies evidence for one behavior under one set of conditions. It does not by itself make the patch trustworthy: the test may be incomplete, another path may regress, or the agent may have exceeded its authority. Trust therefore applies to the whole system, including the model, harness, tools, context, and operating controls.

The later evaluation chapters develop this evidence in stages: Chapter 47 for what a static score measures, Chapter 48 for uncertainty in observed differences, Chapter 51 for support behind generated claims, Chapter 52 for multi-step trajectories, and Chapter 55 for constraining capable systems. Chapter 76 connects capability and resource use to a price that a deployment can sustain.

How to read from here

The next chapter, Chapter 2, places these mechanisms in the full structure of the book. From there, readers can follow the lifecycle in order or enter at a current engineering problem and use the cross-references to recover its dependencies.

Two distinctions should remain after this first pass. Model development and request execution are connected timelines, not one journey. Capability, efficiency, and trust must be evaluated for the complete deployed system, not inferred from model architecture alone. The rest of the book develops the mechanisms behind those claims.

Further reading