Part I: Base Model Formation

"The loss scales as a power-law with model size, dataset size, and the amount of compute used for training."

Jared Kaplan et al., "Scaling Laws for Neural Language Models"

The map from Part 0 becomes material here. Before there is an assistant, a tool call, or a product surface, someone has already chosen how much compute to spend, what text counts as training data, how that text becomes tokens (the integer pieces text is cut into, Chapter 7), what transformer block, the repeated layer that turns a sequence into computation, will process those tokens, and how the run will survive long enough to become a model. These choices look preliminary only from far away. In practice, they set the shape of everything that follows.

Part I is the formation of the base model, where later freedoms are bought and later constraints are locked in. Chapter 5 explains why a training run is a bet made before the model exists. Chapter 6 and Chapter 7 show why the raw web is never the training set, and why the vocabulary is not a cosmetic detail. The blocks that turn sequence into computation are the work of Chapter 8 and Chapter 9: the dense path, the sparse path, and the long-context alternatives that keep returning under new names. Chapter 10 then brings the abstraction back to the cluster: precision, parallelism, communication, checkpoints, and the many ways a run can fail without producing anything useful. Chapter 11 closes the part by showing what happens after the first broad pass over the web but before post-training: high-quality annealing, domain bridges, specialist continuation, and context extension, all of them changes to the data mix and training schedule rather than to the objective.

The useful question is not merely how pre-training works, but which decisions can still be changed later and which have already become part of the model's body. Serving, adaptation, evaluation, and safety all inherit answers from here. Some of those answers are mathematical, some are engineering compromises, and some are scars left by the hardware.