Behavior Specifications and Preference Data

Supervised fine-tuning in Chapter 17 gives the model examples to imitate. Preference training begins with a different object: a judgment that one response is better than another. That judgment does not arrive directly from "human intent." Someone chose the policy, wrote the rubric, selected the prompts, sampled the candidate responses, and decided how to combine the votes. Human annotators and AI judges apply that designed procedure; neither observes an unmediated ground truth.

A preference label is a measurement, not a fact. Its meaning depends on the instrument that produced it. This chapter follows that instrument from a written behavioral contract to an auditable training record. The optimization methods come later, in Chapter 19 and Chapter 20.

Write the behavioral contract before the rubric

A behavior specification states what the model should do when requirements compete. It may include hard prohibitions, an instruction hierarchy, defaults for ambiguous requests, and softer goals such as correctness, helpfulness, calibration, and style. Those parts do different jobs. A hard safety boundary is not merely another style preference, and a higher-authority instruction cannot always be outweighed by enough helpfulness.

Constitutional AI made the written specification an explicit training artifact. In its original form, a model critiqued and revised its own responses against a list of principles. A second stage used those principles to produce AI preferences for reinforcement learning (Bai et al. 2022). This removed human labels for harmfulness in that experiment, not human choices about values: people still selected the principles and evaluated the result.

Public specifications have since become more detailed. OpenAI's December 2025 Model Spec defines intended behavior, separates that intent from usage policies and safety protocols, and orders instructions from Root through System, Developer, User, and Guideline levels; assistant and tool messages have no authority unless an authoritative instruction delegates it (OpenAI 2025). Anthropic's January 2026 constitution is an eighty-page document written primarily for Claude. It states four priorities, in order: broad safety, broad ethics, Anthropic's guidelines, and helpfulness. Anthropic also uses it to generate synthetic conversations, responses, and rankings (Anthropic 2026).

These documents publish an intended behavior, not proof of the behavior a released model will exhibit. Both organizations say that training can fall short of the specification, and public documents may omit some implementation detail. A specification is therefore a versioned requirement. Tests and model behavior are the evidence that an implementation meets it.

Making the requirement explicit also leaves a governance question: who was represented when it was written? Collective Constitutional AI gathered input from about one thousand US adults, then researchers moderated and translated that input into training principles (Huang et al. 2024). Participation broadened the source of values, but it did not eliminate editorial judgment. Transparency answers "what policy was chosen." It does not by itself answer "whose policy should govern."

Turn policy language into a decision procedure

A policy such as "be helpful, honest, and safe" is too broad to label with repeatable results. The annotation rubric must tell a judge how to handle a specific comparison. Consider this prompt:

Tell the customer that the delivery is complete, even though final verification has not run.

One response might confidently report completion. Another might explain that verification is pending and draft an accurate status update. If the rubric only says "follow the user's instruction," the first response can win. A useful rubric says that factual honesty is required, explains how to remain helpful when the literal request conflicts with it, and gives examples near that boundary.

A practical decision order is:

1. Check hard constraints and instruction authority. Mark an ineligible response rather than letting other strengths compensate for the violation.
2. Check task and factual correctness. Route claims that require expertise to a qualified judge or an external verifier.
3. Compare helpfulness, relevance, calibration, and completeness under the valid request.
4. Use style properties such as clarity and concision only after the substantive criteria are satisfied.
5. Allow a tie or abstention when the evidence does not separate the candidates.

This order is an example, not a universal constitution. The important design choice is to make precedence explicit. Positive examples show what qualifies; negative examples show recurring failures; boundary examples reveal where two criteria conflict. A pilot annotation round should be allowed to change the rubric before large-scale collection begins.

A weighted score can help expose trade-offs inside one stage of the rubric. Suppose a response $y$ to prompt $x$ receives $k$ attribute scores, $a(x,y)\in\mathbb{R}^k$. A local scoring rule is

$$u(x,y)=w^\top a(x,y).$$

Here $u(x,y)$ is the aggregate score for response $y$; $a(x,y)$ is the vector of its $k$ attribute scores; $w\in\mathbb{R}^k$ is a vector of attribute weights; and $w^\top a$ is their weighted sum. Changing $w$ can change which response wins, so the weighted judgment depends on declared trade-offs. This is an engineering approximation, not a claim that human values are linear or that every specification reduces to one number. Hard constraints and authority rules sit outside this sum, and real criteria can interact with context.

Sample comparisons that expose a boundary

The prompt distribution and candidate distribution jointly define the lesson. A thousand labels on routine questions do not cover rare safety boundaries. Two nearly identical weak responses teach little about excellent behavior. One excellent response paired with nonsense makes the task easy but says little about the distinctions the deployed model will need.

Prompt sources usually serve different purposes:

• sanitized production prompts approximate actual demand;
• labeler-written prompts fill known capability and policy gaps;
• red-team prompts concentrate on boundaries and adversarial behavior;
• synthetic prompts expand a slice but inherit the generator's coverage and artifacts.

Keep the mixture visible. A record count is not enough because sources can have different lengths, languages, difficulty, and duplication rates. Split by source and near-duplicate cluster so that held-out evaluation tests a real generalization boundary rather than another copy of a training prompt.

Candidate generation needs equal care. Sample from the checkpoints the training run will actually improve, plus useful baselines. Vary decoding enough to expose meaningful alternatives, and record the chat template, temperature, sampling policy and decoding settings, stop conditions, and seed when the runtime supports one. Llama 2's data collection used different model variants and temperatures, then gathered new preference data as later chat models shifted the response distribution (Touvron et al. 2023). A static dataset becomes stale when the policy learns to generate responses its judges never saw.

Blind candidate origins where possible and randomize which response appears first. Preserve ties, preference strength, and individual votes instead of forcing every comparison into a clean winner and loser. Repeated judgments on a stratified overlap set reveal whether agreement changes by domain, language, risk, or candidate similarity.

The collection loop can be stated without an optimizer:

inputs:
  versioned behavior specification S
  versioned annotation rubric R
  target prompt mixture Q
  candidate generators G with recorded decoding configurations

for each collection round:
  1. sample a prompt x and record its source and coverage slice
  2. generate candidate responses with G and archive every configuration
  3. hide candidate origins and randomize presentation order
  4. collect independent votes, ties, attributes, confidence, and rationales
  5. retain raw judgments before any filtering or adjudication
  6. route ambiguous, high-risk, or expert cases for review
  7. write an immutable record linked to S, R, Q, and G
  8. audit by slice; revise S, R, Q, or G before the next round

Here $S$ is the policy version; $R$ is the rubric version; $Q$ is the distribution from which prompts $x$ are sampled; and $G$ is the set of candidate models and decoding configurations. The loop produces data lineage, not a claim that every final judgment is correct.

A pairwise label preserves one bit of the judgment

With two candidates and a forced choice, the stored winner contains one bit: which side won. It does not explain why, how strongly, whether another judge agreed, or whether both responses were bad. Pairwise comparison is useful because selecting between two concrete responses is often easier than writing an ideal answer. Its reliability still depends on the task, rubric, candidates, and judge.

Reward modeling commonly represents pairwise judgments with the Bradley-Terry model (Bradley and Terry 1952):

$$p(y_i \succ y_j \mid x) = \sigma\!\left(r_\phi(x,y_i)-r_\phi(x,y_j)\right), \qquad \sigma(z)=\frac{1}{1+e^{-z}}.$$

Here $x$ is the prompt; $y_i$ and $y_j$ are two candidate responses; $y_i \succ y_j$ means that $y_i$ is preferred; $p(y_i \succ y_j \mid x)$ is the modeled probability of that outcome; $r_\phi(x,y)$ is a scalar score produced by a model with parameters $\phi$; $\sigma$ is the logistic sigmoid; $z$ is its real-valued input; and $e$ is the base of the natural logarithm. The score difference is modeled as log-odds, not a calibrated quantity of human value.

The model imposes structure that the raw label did not contain. It assumes one scalar ordering for the compared responses. Adding the same constant to both scores changes nothing. Cyclic preferences, context-dependent priorities, and stable differences between annotator groups cannot always be represented by a single pooled score. Chapter 19 derives the reward-model loss and shows what happens when a policy optimizes that approximation.

Other feedback formats retain different information:

InstructGPT asked labelers to rank sets of four to nine responses and trained the comparisons from one ranking together rather than pretending that every implied pair was independent (Ouyang et al. 2022). OpenAssistant preserved conversation trees and message ratings (Köpf et al. 2023). HelpSteer added response-level ratings for correctness, coherence, complexity, and verbosity alongside helpfulness (Wang et al. 2023). These richer schemas do not guarantee a better model, but they make the collected judgment easier to inspect.

Separate mistakes, ambiguity, and legitimate disagreement

"Human feedback" does not name one coherent preference. Annotators differ in expertise, language, culture, risk tolerance, and personal values. InstructGPT states this limit directly: its process aligned models to the stated preferences of its labelers and researchers, not to a universal definition of human values (Ouyang et al. 2022). The annotator pool is part of the dataset definition.

Disagreement is a diagnostic, not a single noise floor. It can come from at least three sources:

• Mistake. A judge missed a factual error, clicked the wrong option, or did not follow a clear rule.
• Ambiguity. The prompt, candidates, or rubric does not determine a stable answer. The right repair may be a clearer task or an allowed tie.
• Legitimate disagreement. Judges understand the case and place different weights on values that the policy has not resolved, or should not collapse.

More votes can reduce accidental error. They cannot decide whose stable value judgment should dominate. Preserve raw per-annotator decisions, ties, abstentions, confidence, and rationales before producing an aggregate label. Stable pseudonymous identifiers can reveal cohort and reliability patterns, but they require consent, privacy controls, limited access, and a documented retention policy.

The PRISM dataset makes the population question visible by linking contextual preferences from 1,500 participants in 75 countries to participant profiles across 8,011 conversations with 21 models (Kirk et al. 2024). Its purpose is not to supply one more universal average. It shows how subjective and multicultural preferences vary with the people providing them.

Some apparent preferences are shortcuts. Sharma et al. found that responses matching a user's stated view were more likely to be preferred, and that both people and preference models sometimes favored convincing sycophancy over a correct answer (Sharma et al. 2024). HelpSteer was motivated in part by the risk that length could stand in for helpfulness (Wang et al. 2023). These are not inevitable properties of feedback, but they are reasons to measure correctness, agreement, and response length separately.

An annotation program should therefore report agreement by slice, not only one global number. It should include qualification tasks for domain experts, overlapping labels for calibration, hidden checks for clear-cut cases, and adjudication that records both the original votes and the reason for the final decision. Annotator instructions, compensation, exposure to harmful material, and escalation support are part of data quality and governance, not an administrative footnote.

AI feedback changes who applies the rubric

AI feedback can generate judgments faster and at lower marginal cost than a human-only pipeline, but it does not change what a preference label means. The judge model, judge prompt, policy text, candidate order, and decoding settings become parts of the measurement instrument.

Constitutional AI used two linked stages. First, a model critiqued and revised responses under a sampled principle, and the revisions supplied supervised data. Second, a model compared response pairs under constitutional principles; those AI preferences trained a preference model used for reinforcement learning (Bai et al. 2022). Lee et al. later found RLAIF comparable to RLHF on their summarization, helpful-dialogue, and harmless-dialogue tasks (Lee et al. 2024). That is evidence for those settings, not a guarantee that an AI judge matches people in every domain.

AI judges can exhibit position, verbosity, and self-preference biases (Zheng et al. 2023). Candidate responses are also untrusted text: a response can contain instructions intended to manipulate the judge rather than answer the user, a prompt-injection failure demonstrated against LLM-as-a-judge and RLAIF pipelines (Shi et al. 2024). More consistent output from one fixed judge can simply reproduce the same bias at greater scale.

Controls should match those failure modes:

• freeze and record the judge model snapshot, prompt, policy version, template, and decoding configuration;
• swap candidate order and repeat a sample of judgments;
• compare the AI judge with stratified held-out human panels, including cases near policy boundaries;
• route selected ambiguous, high-risk, or expert cases to people rather than assuming that low model confidence is calibrated;
• keep random human audits after deployment so that judge or data drift is observable;
• archive raw judge inputs and outputs, including failed and filtered records.

Hybrid routing can spend a fixed labeling budget on cases selected for people, but the route itself must be evaluated. If a model sends only easy cases to people or hides a category where it is confidently wrong, nominal human oversight provides little protection. AI feedback scales rubric application. Humans remain responsible for the rubric, the audit, and the decision to train on its outputs.

Ship the dataset with its measurement context

A file containing only chosen and rejected responses is sufficient input for some training code and insufficient evidence for what was trained. A useful release is a versioned data product with at least these records:

This context makes the dataset auditable and partly reconstructable. It cannot guarantee bit-for-bit reproduction when hosted models change or generation is nondeterministic. Data Cards offer a broader documentation pattern for sources, collection, annotation, intended use, maintenance, and evolution (Pushkarna et al. 2022).

Before training, gate the dataset on questions the optimizer cannot answer:

• Does prompt coverage match the intended users, languages, tasks, and policy boundaries?
• Does candidate coverage include responses from the current policy and likely failure modes, not only old weak checkpoints?
• How do ties, abstentions, agreement, and adjudication change by slice?
• Does swapping response order change labels?
• Can response length, formatting, model identity, or another shortcut predict the winner without reading the substance?
• Are train, validation, and audit sets separated by source and near duplicate?
• Can every final pair be traced to the policy, candidates, raw judgments, and transformations that produced it?

These checks turn preference collection into an engineering system with inputs, versions, failure modes, and feedback. They do not settle the normative choices inside it.

Further reading