Part III: Post-Training: Adaptation, Preference, and Alignment

"Making language models bigger does not inherently make them better at following a user's intent."

Long Ouyang et al., "Training language models to follow instructions with human feedback"

Part I produced a base model that learns the shape of text; Part II followed the generative and multimodal architectures that branch off the same foundations. Post-training is the stage after pre-training: where pre-training learned the shape of text from raw data, post-training adapts that base model toward the behavior a product actually needs. This part returns to that base model and the gap it leaves: it does not yet know which behavior a product wants from it, or which refusal a policy requires. It has no way to judge which authority wins when two of them conflict, and given a choice between answers, it cannot tell which a human will prefer. The weights already contain a great deal of capability, but capability by itself is not a usable interface. It must be pointed, constrained, ranked, tested, and sometimes taught to prefer one kind of answer over another.

Chapter 17 starts with the most direct intervention: demonstrations that reshape behavior, often through small adapters rather than a full rewrite of the model. Chapter 18 then asks what the preference signal is supposed to mean, tracing behavior specifications, constitutions, annotator protocols, and multi-attribute preference data before any optimizer sees them. With Chapter 19 the reward-modeling loop appears, where the system learns from comparisons and a reward proxy, a learned model that scores answers in place of a human, rather than from a single correct answer. Chapter 20 shows why that loop can be algebraically folded into direct preference objectives, and why the many variants differ less by slogan than by the balances they choose. Learned rewards split from checkable ones in Chapter 21, the hinge that connects this part to the reasoning machinery in Chapter 28. Chapter 22 follows the same post-training stack into refusals, instruction hierarchy, and deliberation over written policy. Chapter 23 closes the circle by letting models produce, filter, judge, and improve on data that no longer came directly from a human labeler.

As this part proceeds, the word alignment, shaping the model's behavior to be helpful and safe rather than merely accurate, narrows. It comes to name not a model that is good in every sense but the choice of a training signal, the acceptance of its proxy failures, and a decision about where oversight enters the loop. The central cost is no longer only compute. It is trust in the signal that tells the model what better means.