Runtime Safety: Guardrails and Moderation

Alignment in Chapter 19 taught the model to refuse a harmful request most of the time. But "most of the time" is not a guarantee, and a deployed system cannot rely on it. A served system answers untrusted input on every request, the policy it must enforce changes faster than the model can be retrained, and the next chapter shows the refusal itself can be jailbroken, coaxed into answering anyway (Chapter 58). So production systems add a second safety layer that runs at inference time and does not depend on the model behaving. That layer exists because an aligned model's own refusal is necessary but not sufficient: input and output have to be screened separately, moderation has evolved from blocklists into policy-conditioned classifiers, tool-using agents reopen the problem in its hardest form through indirect prompt injection, and the serving layer's streaming decision reaches up to dictate how the guardrail is built.

Why the aligned model is not enough

The oversight chapter drew a line between teaching a model the right behavior and bounding its behavior from outside (Chapter 55). Runtime safety is that same line drawn one layer down, at serving time. The reason a built-in refusal is not enough is structural, not a matter of insufficient training. The model's compliance is probabilistic: it refuses with high probability, not certainty, and a system serving billions of requests meets the tail. And the policy keeps shifting underneath it. A platform's content rules, a regulator's new prohibition, a customer's domain restriction all change on a timescale of days, while retraining the model to encode them costs weeks and risks regressing everything else. On top of all that, the model's safety can be actively attacked, which the next chapter takes up in full.

The standard response is the one control theory and security both reach for: a control that is external to the thing it controls. A guardrail is a separate, usually smaller, classifier placed in the request path, cheap enough to run on every call, and updated independently of the model it guards.

The shape of a guardrail

The guardrail screens twice, because the two screens catch different failures.

Input screening catches a disallowed request before the model spends a single token on it, which is both a safety win and a cost win: the cheapest harmful generation is the one never produced. Output screening catches what input screening cannot, because a benign-looking prompt can still elicit a disallowed completion, and because the model can be steered by content it retrieved or a tool returned mid-generation. The two screens are independent classifiers, and the practical default is to fail closed: if the guard errors or times out, the request is refused rather than passed through.

Evolution: from blocklists to policy as input

The lineage runs from crude to programmable. The earliest filters were keyword blocklists, trivially bypassed by spelling and trivially wrong on benign text. Learned toxicity classifiers replaced them with a model of the category, the line whose modern form is Jigsaw's Perspective API, which dropped per-language token vocabularies for a single character-level transformer robust to spelling, code-switching, and obfuscation (Lees et al. 2022). The next step put a language model in the moderator's seat: OpenAI's moderation endpoint is a classifier trained over a designed taxonomy of harm with an active-learning pipeline to keep catching the rare violations production traffic surfaces, more robust to paraphrase than a keyword list and maintained rather than fixed (Markov et al. 2023).

The decisive shift was making the policy an input rather than a baked-in category. Llama Guard is fine-tuned to take a safety taxonomy in its prompt and classify a conversation against that taxonomy, with separate prompt and response modes, so the same guard model serves many policies and a policy change is a prompt change, not a retraining (Inan et al. 2023). Its later versions align to the MLCommons hazard taxonomy, a shared vocabulary of harm categories that lets guards and benchmarks agree on what they are measuring (Vidgen et al. 2024), and the line now spans modalities: Llama Guard 4 (2025) classifies text and images against that taxonomy with a single model (Meta AI 2025). Variations on the guard model fill out the design space: ShieldGemma ships a range of sizes with calibrated probabilities so an operator can pick a threshold (Zeng et al. 2024), and WildGuard folds three jobs into one model, scoring prompt harm, response harm, and whether the model refused, the last because over-refusal is itself a failure worth measuring (Han et al. 2024). A parallel line added control above the single classification: programmable rails that constrain the dialogue itself, routing or rephrasing or refusing according to declared flows rather than a single yes-or-no (Rebedea et al. 2023). The arc is from a fixed filter toward a configurable policy engine that sits beside the model.

The injected agent: runtime safety's hardest case

The guardrail above assumes the dangerous content arrives in the user's turn. A tool-using agent breaks that assumption. The model of Chapter 37 is trained to emit tool calls, and the harness of Chapter 41 executes them and feeds the results back into the context, so the agent reads web pages, documents, emails, and tool output as data, and a transformer conditions on its whole context without a reliable way to tell which spans are trusted instructions and which are untrusted data. That is the structural root of prompt injection, named by analogy to SQL injection (Willison 2022). Direct injection is the user typing "ignore the above"; the dangerous case for agents is indirect injection, where the malicious instruction is hidden in content the agent retrieves, demonstrated against real applications by Greshake et al., who showed a planted payload in a web page could hijack an LLM-integrated assistant (Greshake et al. 2023). The precondition for catastrophe is a lethal trifecta: an agent that at once has access to private data, exposure to untrusted content, and a channel to communicate outward can be made to route the private data to an attacker, and removing any one leg breaks the exfiltration path (Willison 2025).

The defenses are real but each is partial. The instruction hierarchy trains the model to rank its sources, system above user above tool output, and obey a lower-privilege instruction only when it aligns with a higher one (Wallace et al. 2024); spotlighting marks untrusted spans so the model treats them as inert (Hines et al. 2024). Both raise the bar but remain learned priors a clever framing can defeat. The one approach that offers a guarantee rather than a prior is structural: the dual-LLM pattern, generalized by CaMeL, runs a privileged model that plans and calls tools but never sees raw untrusted content, and a quarantined model that processes untrusted content but holds no tools, with a capability layer that fixes the control and data flow from the trusted query alone so injected data cannot redirect it (Debenedetti et al. 2025). It buys provable security at a utility cost, on the AgentDojo benchmark solving 77 percent of tasks against 84 percent undefended (Debenedetti et al. 2024; Zhan et al. 2024). CaMeL is one member of a pattern language the field has since written down: Beurer-Kellner et al. distill six structural patterns for agents under injection, among them plan-then-execute, the dual LLM, and code-then-execute, each buying a bounded blast radius by giving up some of the agent's generality (Beurer-Kellner et al. 2025). The operational lessons compose into defense in depth: scope each tool to least privilege so a hijack inherits a small blast radius, require human confirmation for irreversible actions, and screen outbound tool arguments and URLs, not just prose, because an injected agent exfiltrates through a crafted request, the data-flow form of the output guard. This content-level screening pairs with a cheaper connection-level destination allowlist, and for a static credential the leak itself can be made inert by egress substitution (Chapter 56).

Jailbreak-robust classifiers

A guard is itself a model with an input channel, so it can be jailbroken like the model it guards, and the frontier of the field is making the guard hard to break. Constitutional Classifiers train input and output classifiers on synthetic data generated from a written constitution, a fixed set of rules spelling out permitted and forbidden content that the classifiers are trained against, augmented with translations and known jailbreak transformations; the output classifier is streaming, a value head (a small scoring head added on top of the model) predicting harmfulness token by token so generation can be halted mid-stream (Sharma et al. 2025). Anthropic reports the classifiers cut a jailbreak success rate from 86 percent on the unguarded model to 4.4 percent, at a 23.7 percent inference overhead and only a 0.38 percent rise in production refusals, and that thousands of hours of red-teaming in a prototype phase found no universal jailbreak, a result that held only for that phase, since a later public demo did surface one. A complementary line, rapid response, observes a handful of real jailbreaks, automatically proliferates variants, and fine-tunes the input guard on them, cutting attack success by more than an order of magnitude from a few examples (Peng et al. 2024). The guard is becoming a trained, adversarially-hardened component in its own right rather than a simple classifier.

Trade-offs

The first trade-off is precision against recall, and it is unforgiving in both directions. A guard tuned to catch everything blocks benign requests, and over-refusal is a real cost: it is the failure users feel, and it pushes them to systems with weaker safety. It is now measured directly, by contrast suites that pair safe prompts a calibrated model should answer with unsafe ones it should refuse (Röttger et al. 2024), and at scale by automatically generated over-refusal benchmarks (Cui et al. 2025). A guard tuned to preserve utility lets harm through. There is no setting that escapes the curve; there is only the choice of operating point, and the right point differs by domain.

The second trade-off is safety against latency, and it is sharpened by the serving layer. Output screening wants the whole generation, but the serving stack streams tokens to the user as they are produced (Chapter 33). A system cannot both stream a token and wait to screen the complete output. It must either buffer the generation and screen it whole, paying latency and giving up the streaming experience, or screen incrementally in chunks, risking that a disallowed prefix reaches the user before the chunk that condemns it, which is exactly why a streaming-aware classifier that scores token by token became worth building.

Implementation

The control flow is small, and its smallness is the point: a guardrail must be cheap enough to run on every request.

def guarded_generate(request, policy):
    verdict = input_guard.classify(request, policy)   # separate small model
    if verdict.blocked:
        return refusal(verdict.category)
    output = model.generate(request)                  # the served model
    verdict = output_guard.classify(output, policy)   # policy passed as input
    if verdict.blocked:
        return replacement(verdict.category)
    return output

Two details carry the design. The policy is passed as an argument, not compiled into the classifier, which is what lets one guard model serve many products and absorb a rule change without retraining (Inan et al. 2023). And the guard is a different model from the one it guards, often much smaller, so that screening every request is affordable and so that a single jailbreak does not compromise both the generator and its monitor at once.

Runtime safety as external control

Capability, at this layer, is mostly something the guardrail spends rather than adds: every block is a refused request, so the discipline is to subtract as little utility as possible while removing the harm, which is the precision against recall curve made concrete. Efficiency explains why the guard is a small, separate model: screening every request is only viable if the screen is cheap, and input screening pays for itself by killing harmful generations before the expensive model runs. Trust is the whole purpose. A built-in refusal asks the user to trust that the model will behave; an external guard that fails closed and is auditable independently of the model is what lets an operator make a claim about the system's behavior that does not rest on the model's good intentions. The guardrail is the runtime instrument of the same control agenda that Chapter 55 argued for at training time.

Further reading