Safety Tuning and Instruction Hierarchy

Instruction following is not enough. A model that grants every user request is unsafe, while one that refuses every risky-looking request is useless. And when a model treats tool output as instruction, it becomes vulnerable to prompt injection, where untrusted input smuggled into that text hijacks the model's own instructions. Safety tuning is the part of post-training that teaches these boundaries before the runtime system ever sees a request.

The training-time layer comes first. Runtime filters, policy engines, and authorization systems appear later in Chapter 57 and Chapter 56. The narrower task here is still foundational: teach the model which behavior is allowed, which instruction has authority, and how to reason over a written policy without turning every borderline request into a refusal.

Safety as a calibrated boundary

A refusal is a classifier decision expressed in natural language. For a prompt $x$, the model chooses among actions such as answer, refuse, ask a clarifying question, transform safe portions, or escalate. A simple constrained objective captures the tension:

$$\max_{\pi}\; \mathbb{E}[\operatorname{helpful}(x,y)] \quad \text{subject to} \quad \mathbb{E}[\operatorname{risk}(x,y)] \le \tau.$$

Here the policy $\pi$ is optimized for helpfulness while the expected risk must stay below the threshold $\tau$. The threshold $\tau$ is a policy choice. Lower it and the model refuses more, reducing harmful completions but increasing over-refusal, where the model declines harmless requests. Raise it and the model becomes more helpful but lets more harmful requests through. RLHF can teach this boundary from comparisons, and constitutional methods can teach it from AI critiques over written principles (Bai et al. 2022). But a boundary learned only from examples is brittle. Attackers search the space that the examples did not cover, and ordinary users suffer when a broad refusal template catches benign cases near a harmful class.

Instruction hierarchy

The second safety problem is authority. A modern model reads many text sources: system prompts, developer messages, user instructions, retrieved documents, tool outputs, web pages, emails, logs, and memory. Some are instructions; others are untrusted data. If the model cannot distinguish them, an attacker can place a malicious instruction inside data and have the model obey it.

Instruction hierarchy makes the authority order explicit. Wallace et al. define and train models to prefer higher-priority instructions over lower-priority conflicting ones: in their scheme the system message outranks user messages, which outrank the model's own earlier outputs, which outrank tool outputs, with developer text folded into the system message (Wallace et al. 2024). OpenAI's Model Spec expands the hierarchy in product terms, separating the developer into a level of its own between system and user and adding root and guideline levels (OpenAI 2025).

The training example is a conflict pair. Given context $c$ containing two conflicting instructions, choose response $y^+$ that follows the higher authority and reject response $y^-$ that follows the lower one:

$$\mathcal{L}_{\text{hier}} = -\log \sigma\!\left(r(c,y^+) - r(c,y^-)\right).$$

This is the same pairwise machinery as Chapter 19, but the semantics of the preference are different. The label is not "which answer is nicer." It is "which answer respects the authority order." That difference matters because it turns safety from a list of forbidden topics into a source-sensitive behavior. The model learns that the same words have different force depending on where they came from.

Deliberation over policy

Safety examples alone do not cover the boundary. Deliberative Alignment adds a different ingredient: teach the model the written safety specification itself and train it to reason over that specification before answering (Guan et al. 2024). It trains the model to recall and apply policy text at inference time, rather than to pattern-match a refusal. This is a change in what the model internalizes. Instead of only learning surface refusal patterns, it learns to recall policy text, apply it to the request, and choose a response type. Constitutional AI made the same pattern visible earlier through self-critique and revision against written principles (Bai et al. 2022); Deliberative Alignment makes the policy reasoning more direct.

The benefit is generalization. A model trained only on prompt-to-refusal pairs learns local neighborhoods of danger. A model trained to reason over policy has a chance to handle an unseen request by mapping it to a rule. The risk is that policy reasoning can become theatrical. A model may produce a plausible justification for a decision it already would have made, or memorize policy phrases without improving calibration. The evaluation must therefore test the decision, not the beauty of the explanation.

This is also where behavior specifications from Chapter 18 re-enter the pipeline. A safety-tuned model cannot be better than the policy it was given. If the policy is ambiguous about dual-use biology, privacy, self-harm, or political persuasion, the model will inherit that ambiguity. Written policy does not remove judgment; it relocates judgment into an auditable artifact.

Refusal calibration

The hard practical problem is not making a model refuse. It is making refusal specific. A good refusal boundary has several properties:

• it refuses the harmful transformation, not the topic itself;
• it answers benign adjacent requests instead of treating the whole region as forbidden;
• it gives safe alternatives when those alternatives are genuinely useful;
• it does not let style, roleplay, translation, or encoding move a harmful request outside the boundary;
• it preserves the instruction hierarchy under tool use and retrieval.

These properties pull against one another. Broad refusals reduce false negatives but raise false positives. Narrow refusals preserve usefulness but expose more edge cases. Better training tries to move the whole frontier: more true harmful requests refused at the same benign-refusal rate. But there is still an operating point, and that operating point is a product and governance decision, not a mathematical constant.

The refusal itself is also being retrained. OpenAI's safe-completions method, shipped in GPT-5, replaces the binary refuse-or-comply decision with output-centric training: the model learns to maximize helpfulness subject to the constraint that its own output stays within safety policy, rather than to classify the user's intent and answer or refuse on that classification (Yuan et al. 2025). The target is dual-use prompts, where intent classification fails because the same question serves the safety engineer and the attacker alike. This is the chapter's constrained objective, maximize helpfulness subject to risk below $\tau$, turned into a training recipe and applied to the output rather than the request.

Boundary with runtime safety

Training-time safety is necessary and insufficient. It is necessary because the model's default behavior must already respect policy most of the time. It is insufficient because a deployed system faces fresh attacks, changing policy, tool permissions, jurisdictional differences, and data the model has never seen. Around the model, Chapter 57 adds filters, classifiers, and response gates. Chapter 56 brings capability control, so the model cannot take actions merely because text asked it to. And Chapter 58 studies jailbreaks and the other ways the learned boundary gets probed.

The post-training layer should therefore be read as the model's first line of behavior, not the system's last line of defense.