Behavior Specifications and Preference Data

Chapter 17 showed how a base model becomes an assistant-shaped model by imitating demonstrations. The next question comes before RLHF, DPO, or any optimizer: who decided what the demonstration or preference is trying to express? A pairwise label that says response A is better than response B is not raw human intent. It is the output of a policy document, a rubric, a sampling procedure, an annotator pool, and a data filter. Post-training starts there.

This chapter treats preference data as an engineered measurement system. A behavior specification states the intended behavior; a rubric turns that specification into judgments; annotators or AI judges apply the rubric; and the resulting comparisons become the training signal. If that chain is vague, the model learns the vagueness. If it rewards length, agreement, or confident tone, the optimizer will find those handles.

Before the loss, a policy

The alignment literature often begins with the loss. The operational pipeline begins with a written specification. Anthropic's Constitutional AI made this explicit by replacing much human harmlessness labeling with critique and revision against a short list of principles (Bai et al. 2022). Anthropic's public description of Claude's constitution frames those principles as a way to make values inspectable and editable rather than hidden in a pile of labels (Anthropic 2023). The constitution has since grown into a training artifact in its own right: the January 2026 revision runs to roughly eighty pages, is released under CC0, and is written primarily for Claude itself rather than for the humans labeling its outputs, with a stated priority order of safety and human oversight, then broad ethics, then Anthropic's guidelines, then helpfulness (Anthropic 2026). OpenAI's Model Spec makes the same move in a more product-facing form: it defines intended behavior, separates it from usage policy and safety systems, and orders instruction authority from root and system messages down to developer, user, and guideline-level instructions (OpenAI 2025).

Those documents are upstream training artifacts in their own right. A labeler asked to prefer "helpful and harmless" behavior still needs to know whether a correct but blunt answer beats a warm but vague one, whether a medical request should be answered with caution or refused, and whether a user instruction can override a tool message. The written specification is where those choices become stable enough to label.

A written specification can also do more than stabilize labels. When the named behaviors become the target of reinforcement learning rather than only the rubric a labeler applies, the effect can reach past the cases the rubric covered. OpenAI reports that reinforcement learning toward seven beneficial traits, truthfulness, epistemic humility, metacognitive transparency, corrigibility, risk sensitivity, universal fairness, and concern for human welfare, improved alignment on most of a 53-benchmark held-out suite, carried over from training in a single domain to others, and persisted under adversarial steering and fine-tuning (Jagadeesh et al. 2026). The specification remains the upstream object; how broadly its effect generalizes is an empirical question the field is now measuring rather than a property the document guarantees.

The formal object is a utility, even when nobody writes one down. Suppose a response $y$ to prompt $x$ is scored on attributes. Here $a(x,y) \in \mathbb{R}^k$: correctness, helpfulness, safety, calibration, concision, style, or whatever the rubric names. A simple utility is

$$u(x,y) = w^\top a(x,y),$$

where $w$ is the weight vector implied by the specification. This equation is not a claim that human value is linear. It is a discipline for reading labels: when a pair is marked preferred, some implicit weight vector made that trade. Changing the weights changes which answer wins, even if the two answers and their factual content are unchanged.

Pairwise labels and what they can say

The simplest scalable label is a pairwise comparison. Given a prompt $x$, where $y_w$ is the preferred response and $y_l$ is the rejected response, the Bradley-Terry model used by reward modeling (training a model, the reward model, to score a response with a single number that stands in for human judgment) writes the preference probability as

$$P(y_w \succ y_l \mid x) = \sigma\!\left(r(x,y_w) - r(x,y_l)\right).$$

Here $r(\cdot)$ is the learned reward score, and $\sigma$ is the sigmoid, an S-curve that maps any real number into $(0,1)$ so the output reads as a probability. In words, the equation turns the gap between the two responses' scores into the probability that a human prefers one over the other: the larger $y_w$'s lead, the more likely it is the chosen response. This is the reason pairwise data flows naturally into Chapter 19 and Chapter 20. It asks a question people can answer more reliably than "write the ideal response": which of these two is better? But the same simplicity is a limitation. A pairwise label says the winner is better than the loser under the current rubric, and not much else. Why it won, whether correctness or tone tipped the decision, whether the annotators even agreed: none of that survives in the label. The comparison is also sensitive to the candidate set, since a response can win because it is good, or merely because the alternative is worse.

That is why modern feedback datasets often add structure around the pair. OpenAssistant Conversations collected conversation trees, message ratings, and multi-lingual human feedback at community scale (Köpf et al. 2023). HelpSteer decomposed judgments into attributes such as helpfulness, correctness, coherence, complexity, and verbosity, explicitly making the reward less dependent on a single scalar (Wang et al. 2023). UltraFeedback used GPT-4 to produce large-scale AI feedback across hundreds of thousands of conversations, then emphasized diversity and bias mitigation because scale alone would only scale the judge's artifacts (Cui et al. 2023).

The design choice turns on how much of the judgment to preserve. Pairwise labels are cheap and robust. Rankings pull more out of a sampled set, attribute labels make the trade-offs auditable, and free-form critiques, expensive as they are to parse, are useful for data repair and constitutional revision. The more structure is preserved, the easier it is to find out later whether a model improved because it became more correct, more verbose, more deferential, or simply better at satisfying the judge.

Annotators are part of the model

Preference labels are often described as "human feedback," as if the human were a single, stable source of truth. In practice the annotator pool is part of the modeling assumption. Labelers differ in domain knowledge, risk tolerance, language, political culture, patience, and susceptibility to confident prose. A non-expert pool labeling a task that demands expertise yields a reward model that learns non-expert preference. Guidelines that collapse helpfulness into agreement breed sycophancy, the habit of telling the user what they want to hear. And if long answers tend to read as more careful, a reward trained on those judgments will acquire verbosity bias, a standing preference for length itself.

Three quantities matter operationally.

• Disagreement. The disagreement rate is the noise floor. When two trained annotators cannot agree under the same rubric, no reward model should be expected to learn a sharper boundary without adding another signal.
• Coverage. The prompt distribution and candidate models define what the annotators ever see. A reward model trained on safe, ordinary responses has little reason to understand adversarial prompts or rare policy boundaries.
• Provenance. Labels need versioned guidelines, annotator metadata, model versions, and sampling settings. Without them, a later behavior change cannot be traced to the model, the data, or the rubric.

Chapter 49 treats human evaluation as an instrument. The same principle applies here, except the instrument is upstream of training rather than downstream of release. A reward model is a frozen copy of that instrument. Once it is optimized against, its biases are no longer measurement error. They become behavior.

AI feedback scales the labeler and inherits its risks

RLAIF (reinforcement learning from AI feedback) and constitutional pipelines replace part of the human labeling process with a model judge (Bai et al. 2022). The gain is clear: AI judges are cheap, consistent, multilingual, and available at the volume needed for repeated post-training. They can also carry the written specification more directly into the data by critiquing and revising responses against named principles. This is why constitutional methods are more than a cheaper RLHF. They make the specification a first-class artifact in the loop. The 2026 Claude constitution carries that to its end point: written primarily for the model rather than for annotators, it is used directly in training, with Claude generating conversations, responses, and response rankings from it (Anthropic 2026), so the specification now seeds the synthetic training data that Chapter 23 describes.

The risk is just as direct. A model judge is itself a model, with its own training distribution, refusal habits, cultural priors, and blind spots. It may prefer fluent explanations over correct ones, reward the surface form of safety over calibrated risk reduction, or import the policy of its provider into a system that intended a different policy. Worse, once the target model is trained to satisfy that judge, it may learn judge-specific artifacts. This is the same Goodhart pressure that appears later in Chapter 19 (when a measure becomes a target, it stops being a good measure), now moved upstream into the data supply.

The practical answer is hybrid supervision. Humans write and audit the specification, inspect samples where the model judge is uncertain, measure judge bias against held-out human panels, and keep a small stream of fresh human labels to detect drift. AI feedback buys scale. It does not remove the need for accountability over what the feedback means.

The operational form

A usable preference dataset is therefore not just a JSONL file of chosen and rejected responses. It is a versioned bundle:

• the behavior specification and policy version;
• the rubric, including positive and negative examples;
• the prompt source and sampling strategy;
• the candidate model versions and decoding settings;
• the annotator or judge identity class, not necessarily personal identity;
• disagreement, adjudication, and quality-control records;
• known biases, excluded domains, and intended training use.

This metadata is what keeps post-training reproducible. Without it, the later optimizer can be perfect and still optimize the wrong thing.