Safety Tuning and Instruction Hierarchy

Safety tuning has to make two different decisions before a model answers. The first is about output policy: what behavior is allowed for this request? The second uses an instruction hierarchy to decide which instructions control the response when the context disagrees. Confusing them creates two familiar failures. A model may comply with a harmful request or refuse a harmless one, and it may obey attacker-controlled text instead of the application that invoked it.

Training can improve both decisions, but it cannot give the model a security perimeter. Post-training changes the model's behavior, not its permissions. Runtime safety, authorization, secret handling, and tool isolation remain separate system responsibilities.

Two decisions, two error surfaces

The output-policy decision asks whether a candidate response stays within a written safety policy while preserving as much useful help as the policy permits. The authority decision asks which instructions are applicable, then resolves conflicts among them. These decisions interact, but neither can replace the other.

A jailbreak and a prompt injection are related but not identical. A jailbreak tries to bypass the model's learned safety behavior and need not exploit a conflict between message roles. Direct prompt injection puts attacker-controlled text in the user's request to override an application's instructions (Perez and Ribeiro 2022). Indirect prompt injection hides the text in a web page, document, email, or other third-party content that later enters the model's context (Greshake et al. 2023). The first two rows of the table are learned behaviors. The third row requires enforcement outside the model.

Refusal calibration is not binary

Older safety recipes often framed the task as intent classification: comply with a benign request and refuse a malicious one. That framing works for clear cases, but it is brittle for dual-use requests. The same question about a biological protocol or software vulnerability may support defense, education, or abuse. The safe response may be a direct answer, a bounded answer that omits operational detail, a clarification, a refusal of one part, or a safe redirection. Refusal is one action, not the definition of safety.

Two rates expose opposite sides of the boundary. Here $D_H$ is a held-out set of harmful requests and $D_B$ a held-out set of benign requests. Define

$$\operatorname{UCR} =\frac{1}{|D_H|}\sum_{(x_i,y_i)\in D_H}\mathbf{1}[V(x_i,y_i)=1],$$

where UCR is the unsafe-compliance rate, $x_i$ is request $i$, $y_i$ is the model's response, $V(x_i,y_i)=1$ means that the response violates the safety policy, $|D_H|$ is the number of harmful test requests, and $\mathbf{1}[\cdot]$ is the indicator function. For benign requests, define

$$\operatorname{BRR} =\frac{1}{|D_B|}\sum_{(x_i,y_i)\in D_B}\mathbf{1}[F(x_i,y_i)=1],$$

where BRR is the benign-refusal rate, $F(x_i,y_i)=1$ means that the response refuses or withholds help that the policy permits, and $|D_B|$ is the number of benign test requests. A lower UCR is not a success if BRR rises until the product becomes unusable.

An average can hide the failure that matters. Report these rates by policy category, language, encoding, conversation length, and tool path. Also report the severity and actionability of unsafe outputs. A short response that avoids a refusal phrase but supplies no useful harmful information should not count as a successful attack.

Safe-completions training moves the target away from a hard comply-or-refuse label and toward the safety of the output. Yuan et al. train direct answers, bounded safe completions, and refusals with redirection, then use learned safety and helpfulness scores during reinforcement learning (Yuan et al. 2025). One disclosed reward is

$$r_i=h_i s_i,$$

where $h_i\in[0,1]$ is the learned helpfulness score for response $i$, $s_i\in[0,1]$ is its learned safety score, and $r_i$ is the resulting reward. Multiplication discourages both unsafe detail and empty but safe answers. It is still a learned proxy, not a hard constraint. The paper reports controlled and production gains, especially on dual-use prompts; those results do not establish that every policy, model, or adaptive attack receives the same trade-off.

Instruction hierarchy resolves applicable instructions

Instruction hierarchy is a conflict rule, not a command to ignore every lower-level message. Lower-authority instructions should be followed when they are applicable and do not conflict with higher-authority constraints. At the same level, a later instruction may supersede an earlier one. Wallace et al. introduced targeted data generation for this behavior and showed improved robustness in experiments on GPT-3.5 (Wallace et al. 2024).

The exact ladder is a product convention rather than a law of language models. In the December 2025 OpenAI Model Spec snapshot, the stated order is Root > System > Developer > User > Guideline, followed by No Authority (OpenAI 2025). Assistant messages, tool outputs, and quoted or retrieved text have no authority by default in that specification. A higher-authority instruction can delegate authority to such content.

Do not confuse role with trust. A developer message can quote attacker-controlled input, and a tool output can contain trustworthy facts without being allowed to rewrite policy. Conversely, a user can deliberately delegate a bounded task to instructions in a repository file. The system must preserve provenance inside messages, not infer trust from prose style or from the channel alone.

A practical resolver can be stated without pretending that natural-language conflicts are always mechanically decidable:

INPUT: context c, candidate instructions I(c), authority level a(i)
1. Identify candidate instructions in authority-bearing sources.
2. Mark quoted or retrieved text, tool output, and prior assistant text as data by default.
3. Add a data-borne instruction only when an applicable instruction delegates authority to it.
4. Remove candidates outside the task, superseded at the same level, or conflicting with a
   higher applicable instruction.
5. Follow every remaining compatible instruction, including lower-authority instructions.
6. If conflict or delegation is ambiguous and the action has material side effects, ask or escalate.
7. Generate response y, then apply output policy and runtime authorization.

Here $c$ is the complete model context, $I(c)$ is the set of candidate instructions found in that context, $i$ denotes one candidate, $a(i)$ is its authority level, and $y$ is the response. Steps 1 through 6 define the behavioral target. They are not a deterministic parser or a proof that the model will follow the target.

Train aligned and conflicting cases

Hierarchy training needs aligned and conflicting cases. If the dataset contains only attacks, a cheap strategy is to ignore users and tools. Wallace et al. avoid that shortcut with two constructions. Context synthesis creates examples in which lower-level context is relevant and should be followed. Context ignorance creates examples in which misaligned lower-level text should not affect the target response. Their reported recipe uses supervised fine-tuning and reinforcement learning from human feedback; the hierarchy is not defined by one universal pairwise loss.

Evaluation needs the same contrast. IHEval contains 3,538 examples across nine tasks with both aligned and conflicting priorities. Its evaluated models all lost substantial accuracy under conflict, showing that ordinary instruction-following accuracy does not establish hierarchy behavior (Zhang et al. 2025). A model must succeed on four cases:

IH-Challenge later built programmatically graded conflict tasks for reinforcement learning and generated adversarial examples online. On the authors' internal GPT-5-Mini experiment, average hierarchy robustness across 16 evaluations rose from 84.1 to 94.1 percent (Guo et al. 2026). That is useful evidence that targeted training can generalize beyond its simplest tasks. It remains one model family, one training recipe, and a mixture of public and internal evaluations, not a universal security guarantee.

Written policy changes the supervision

Written principles can supply more than a list of refusal examples. Constitutional AI and Deliberative Alignment both use written policy, but they do so differently.

Constitutional AI first generates self-critique and revision, then fine-tunes on the revised responses. Its reinforcement phase uses AI comparisons to train a preference model, a procedure called reinforcement learning from AI feedback (RLAIF) (Bai et al. 2022). Deliberative Alignment instead trains synthetic examples that explicitly reference the safety specification and then applies a specification-aware reinforcement signal (Guan et al. 2024). Neither procedure removes the human choices embedded in the policy, category labels, generators, judges, and evaluation sets.

Evaluate policy reasoning at three observable stages: policy recall, policy application, and final output. The response should identify the relevant rule when an explanation is appropriate, apply it consistently to the facts, and produce a permitted useful answer. A fluent rationale cannot rescue a wrong decision. Conversely, a correct response need not expose private chain-of-thought to demonstrate compliance.

The written policy is the supervised target, not ground truth. Ambiguous, inconsistent, or outdated rules create underspecification. Pretraining priors may make the model behave more sensibly than one ambiguous example, or may pull it away from the intended policy. Version the policy, the training data derived from it, and the evaluation suite together.

Evaluate the failure, not the refusal string

Red teaming searches for failures; it does not certify their absence. A useful loop is:

1. Define the policy taxonomy and threat model.
2. Generate attacks with humans, models, transformations, and tool-mediated scenarios.
3. Score the target's actual harmful usefulness, policy compliance, and task success.
4. Cluster failures and convert selected examples into training or regression data.
5. Keep entire attack families, not only near-duplicate strings, held out for evaluation.
6. Repeat with adaptive attacks that observe previous defenses.

Model-generated attacks and model judges can scale this loop, but they carry their own blind spots (Perez et al. 2022). StrongREJECT found that common jailbreak evaluations often overstated success because bypassing a refusal could also degrade the answer into useless text (Souly et al. 2024). Score whether the response delivers the prohibited capability, not whether it contains a refusal phrase.

The release gate should report at least:

• unsafe-compliance rate and severity by policy category;
• benign-refusal rate on contrast sets such as XSTest (Röttger et al. 2024);
• safe-completion helpfulness and ordinary task quality;
• aligned lower-level instruction following and conflicting-instruction accuracy;
• direct and indirect prompt-injection success, prompt leakage, and unsafe tool actions;
• results by language, encoding, multi-turn length, and retrieved-data source;
• held-out attack-family results under a stated adaptive attack budget;
• agreement between automated judges and policy-trained human reviewers.

Do not tune on the release set. Training examples, red-team discoveries, and final evaluation cases need separate lineage so a higher score means more than memorizing the latest attack strings.

Boundary with runtime safety

Safety tuning sets a useful default, but learned behavior is probabilistic. Runtime classifiers are usually probabilistic too. A hard boundary exists only where the system uses deterministic enforcement under an explicit threat model.

An application should keep credentials out of the model context, issue least-privilege tool capabilities, validate tool arguments, require approval for consequential side effects, and record provenance across retrieval and tool calls. Those controls remain effective even when the model misreads an instruction. Chapter 57 develops detection and gating, Chapter 56 develops capability enforcement, and Chapter 58 develops attack evaluation.