Summary

The first substantive part followed base-model formation before the model becomes a product: how scale is budgeted and data selected, what turns bytes into tokens, the way Transformer variants arrange computation, and what lets a training run survive the cluster. The mid-training chapter then showed that the path from broad pretraining to post-training is no longer a blank space; it is where quality, domain mixture, specialization, and context length are adjusted before behavior is explicitly shaped.

The concern running through the part is irreversibility. Data mixture, vocabulary, architecture, precision, parallelism, and continuation curriculum all look like preparation, but they become inherited constraints. Later serving, adaptation, evaluation, and safety work can compensate for some of those choices, yet it cannot freely rewrite them.

Base-model formation is infrastructure, not background. The open question is how much room remains for scaling once natural corpora thin out and hardware economics tighten. Higher-quality data, synthetic data, and mid-training may stretch the curve, but the part leaves that as a constraint to be tested rather than a promise to assume. This also explains why Part II steps sideways before the book returns to post-training: Part I showed how text is organized into token sequences and how a base model inherits that order. Part II asks what happens when the object is an image, a sound, a video, or embodied experience, where the system must invent an order that can be trained, sampled, cached, and evaluated. The connection is not a list of media formats after text; it is the same infrastructure question applied to objects that do not arrive as natural strings.