Runtime Safety: Guardrails and Moderation

Training can make harmful behavior less likely, but deployment needs a stronger statement than "the model usually refuses." An aligned model's own refusal is necessary but not sufficient. Every request brings new input, product policy changes without a model retrain, and adversaries deliberately search for cases the training did not cover. Input and output have to be screened separately, but screening is only one part of the answer. Policy-conditioned classifiers can detect evidence of harm; they cannot authorize a payment, isolate a process, or prove what an external API did.

Runtime safety is the request-time contract that detects evidence, lets policy decide, makes an enforcement point apply that decision, contains what still goes wrong, and records what happened. For a tool-using system, its central rule is simple: the model may propose; only trusted code may authorize and commit. That rule connects the external control discussed in Chapter 55 to the prompt-injection and streaming problems in this chapter. Indirect prompt injection makes provenance part of every decision, and the serving layer's streaming decision reaches up into the safety design.

Five jobs, not one guard model

The word guardrail often hides several different mechanisms. Keeping their jobs separate prevents a detector's uncertain prediction from being mistaken for a system guarantee.

1. A detector extracts signals: moderation scores, malware findings, secret matches, suspicious destinations, or anomalous tool arguments.
2. A policy decision combines those signals with authenticated subject, action, resource, tenant, data classification, and policy revision.
3. A policy enforcement point allows, blocks, rewrites, challenges, or delays the exact operation.
4. Containment limits the damage if detection and policy both miss: narrow credentials, a sandbox, quotas, and network egress controls.
5. Evidence links the request, decision, attempted effect, and observed result so an operator can test and investigate the system.

A score is not a decision. A decision is not enforcement. A blocked request is not proof that no side effect occurred. Those distinctions already mattered for authorization in Chapter 56; runtime safety applies them to model text, retrieved data, generated code, and tool calls.

Input moderation can stop a disallowed request before generation. Output moderation covers cases that are visible only after generation, including a benign request followed by a harmful answer. Neither screen sees every relevant fact. A retrieved document can alter the model after input screening, and a tool can turn an apparently harmless string into an irreversible effect after output screening. The placement and scope of each check therefore matter as much as the classifier.

From a detector score to a policy decision

Modern moderation systems learn categories rather than matching only keywords. Production examples combine a documented taxonomy, carefully reviewed labels, active learning, and synthetic or adversarial data (Markov et al. 2023). Models such as Llama Guard accept a taxonomy with the conversation and support separate prompt and response classification (Inan et al. 2023). This is useful configuration, not arbitrary programmability. Policy text does not make an unsupported category reliable, and a renamed category does not create the data needed to evaluate it.

Treat the policy as a versioned policy bundle with a declared schema, supported categories, thresholds, and failure behavior. Require schema validation, offline evaluation before rollout, a staged release, and rollback. A policy prompt is untrusted configuration until validated: malformed or injected text must not silently change which policy is active.

The decision also needs more than one threshold. It must be able to allow, review, or block. Let $x$ be the content being evaluated; let $k$ identify a harm category; let $s_k(x)$ be the detector's score for that category; and let $v$ identify the policy revision. For each category, policy $v$ sets a lower threshold $\tau_{\mathrm{allow},k}$ and an upper threshold $\tau_{\mathrm{block},k}$, with the lower threshold no greater than the upper one. Read the three branches separately:

$$\begin{aligned} D_v(x,k) &= \mathrm{allow}, \\ s_k(x) &\le \tau_{\mathrm{allow},k}, \\ D_v(x,k) &= \mathrm{review}, \\ \tau_{\mathrm{allow},k} &< s_k(x), \\ s_k(x) &< \tau_{\mathrm{block},k}, \\ D_v(x,k) &= \mathrm{block}, \\ s_k(x) &\ge \tau_{\mathrm{block},k}. \end{aligned}$$

Here $D_v(x,k)$ is the decision under revision $v$. The middle region is an explicit abstain path: request a human decision, route to a safer model, ask for clarification, or disable tools. It is not an instruction to hide uncertain cases inside "allow." Selective classification makes this coverage-versus-risk choice explicit (Geifman and El-Yaniv 2017).

An estimated score is not automatically a probability. If the product presents or reasons about it as one, it must be calibrated on traffic that represents the deployment (Guo et al. 2017). A false negative allows a violating item; a false positive blocks a compliant one. Recall is the fraction of violating items caught, while precision is the fraction of flagged items that truly violate the policy. Precision depends on the base rate. If violations are rare, false positives can dominate the review queue even when recall and the false-positive rate look good on a balanced benchmark.

Calibration, false-positive rate, false-negative rate, and review rate should be reported per category, per language, and for important user groups, not only as a global average. Distribution shift, obfuscation, and adaptive attacks can invalidate an operating point, so release evaluation needs ordinary traffic, contrast cases for over-refusal (Röttger et al. 2024), and a private adversarial test set. Threshold changes are policy changes and deserve the same review and rollback discipline as code.

The failure mode depends on the effect. A timeout while labeling low-risk prose may produce a bounded refusal or a degraded response. Before a privileged tool effect, the rule is to fail closed for privileged effects: commit no external action until every required check returns a valid decision.

Prompt injection changes the threat model

In an ordinary chat request, direct prompt injection puts the conflicting text in the user's message. In an agent, indirect prompt injection puts it in a web page, email, document, database row, image, or tool result that the system later retrieves. The model receives trusted instructions and untrusted data in one context, and untrusted data can influence a privileged decision. Real systems have been compromised in exactly this way (Greshake et al. 2023), while AgentDojo and InjecAgent provide reproducible tasks for measuring attack and defense behavior (Debenedetti et al. 2024; Zhan et al. 2024).

Successful injection is not yet a successful exploit. The injected text must change a model proposal, that proposal must request a harmful operation, and the operation must be accepted by the enforcement point. This decomposition gives the defender several independent places to break the chain.

Learned defenses help at the first step. One approach is an instruction hierarchy (instruction hierarchy), which trains a model to prefer system instructions over user or tool content (Wallace et al. 2024). Spotlighting marks the provenance of untrusted spans (Hines et al. 2024). Moderation and jailbreak classifiers can detect known attack shapes, including transformed examples (Sharma et al. 2025). These learned defenses reduce measured attack success on their evaluation distributions; they do not establish a security boundary. An adaptive input can still be misclassified, and the same model may both interpret the data and decide what it means.

Structural controls address a different question: even if the model is influenced, which control flow and data flow can it change? Plan-then-execute, action selection from a fixed set, context minimization, and dual-model designs separate trusted planning from processing untrusted content (Beurer-Kellner et al. 2025). CaMeL makes this separation explicit with a capability and data-flow layer, and states its security result under a defined threat model rather than for every agent architecture (Debenedetti et al. 2025). The practical design is defense in depth: keep untrusted data out of privileged planning where possible, grant every tool least privilege, and require human confirmation bound to the exact parameters of an irreversible action.

Put a deterministic gate in front of every effect

The harness in Chapter 41 turns generated tokens into API calls, file writes, queries, and code execution. That conversion is the point where runtime safety can become enforceable. A safe effect path performs these checks in trusted code outside the model:

1. Parse the proposal against a versioned, typed tool schema. Reject unknown fields, invalid encodings, ambiguous numbers, and out-of-range values.
2. Normalize the action, resource, and canonical parameters once. Use the same representation for authorization, approval, execution, and audit.
3. Obtain authorization for that exact subject, action, resource, tenant, and context. The model never supplies trusted identity or policy facts.
4. Apply destination policy to every outbound connection. Resolve and validate the DNS answer, connect only to permitted address ranges, and validate every redirect target. A hostname allowlist alone does not stop redirects, DNS rebinding, or a permitted service from exposing an unsafe endpoint.
5. Keep credentials in a secret broker. The model sees an opaque handle, not the secret; the broker injects it only for the authorized destination and action.
6. Before an irreversible operation, persist a write-ahead intent with an idempotency key. Afterward, store the provider's effect receipt or a precise failure. A retry reuses the same key instead of duplicating the effect.

Content moderation can contribute a signal at steps one and four, but it cannot replace any of them. "This URL looks safe" is not network enforcement, just as "this transfer sounds reasonable" is not authorization.

Generated code needs a separate containment boundary. On Linux, namespaces can separate views of processes, mounts, users, and networks; cgroups impose resource limits; and seccomp restricts the system calls a process may attempt. The Linux documentation is careful that seccomp filtering is not itself a sandbox (Linux Kernel Documentation n.d.). Container guidance likewise treats isolation as a collection of configured controls, not a property conferred by an image format (Souppaya et al. 2017). A container is not a virtual machine, and workloads with a stronger attacker or confidentiality requirement may need a dedicated VM or another hardware-backed boundary.

Run generated code without host secrets, without a writable host mount, with resource limits, and with network egress disabled unless a narrow task requires it. If egress is allowed, route it through the same destination and credential broker used for tools. A sandbox limits consequences; it does not decide whether an action is appropriate. It also does not repair a kernel escape, a mounted credential, an overbroad network policy, or a confused downstream service. NIST's generative-AI profile therefore treats monitoring, access controls, incident handling, and testing as complementary risk treatments rather than one universal filter (Autio et al. 2024).

Streaming creates two release boundaries

The serving stack in Chapter 31 emits tokens to reduce perceived latency. Incremental moderation can stop a continuation, but text already sent to the client is already disclosed. A later block cannot retract a secret, a slur, or instructions that the client has received. Chunk size therefore trades latency against the maximum unchecked prefix; it does not remove the trade-off.

The important distinction is between display text and executable output. Streaming text may be acceptable after incremental moderation when the product can tolerate a bounded prefix. A tool call, URL fetch, code execution request, or transaction must remain a proposal until its complete structured value has been buffered, parsed, authorized, and committed by trusted code. Buffer executable outputs even when prose streams. Otherwise a partial stream can become a committed side effect before the system has seen the fields needed to reject it.

An operating contract for runtime safety

The implementation should produce one linked record for each request. At a minimum, record these schema-versioned fields without storing raw credentials or unnecessarily retaining sensitive content:

• request_id, policy_revision, and authenticated subject and tenant;
• input_provenance for user, retrieved, system, and tool-provided content;
• detector_revision_and_scores, decision_and_thresholds, and review outcome;
• model, prompt, retrieval, and tool_schema_revision identifiers;
• authorization_decision_id and canonical_action_and_parameters;
• sandbox_profile, resource limits, and egress_decision including resolved destination;
• approval_binding, expiry, and one-time challenge when approval is required;
• effect_idempotency_key, write-ahead intent, and provider effect_receipt;
• timing, failure_mode, and the enforcement point's final result.

The record makes failures reproducible, but the tests must exercise the actual boundaries. A release suite should include, at least:

• detector error and guard timeout paths for prose and privileged effects;
• policy rollback and mixed-version requests during rollout;
• obfuscation, split encoding, multilingual paraphrase, and benign contrast cases;
• an unknown tool field, duplicate key, non-canonical URL, and oversized argument;
• a redirect to a blocked host, DNS rebinding, private-address resolution, and a permitted host with a forbidden path;
• expired approval, approval replay, changed parameters after approval, and duplicate delivery of the same effect;
• cross-tenant resource references, unavailable authorization, and revoked credentials;
• sandbox escape probes, resource exhaustion, secret-file access, and denied network egress;
• a partial stream that becomes harmful only after several chunks, plus an incomplete tool call that must never execute.

Measure false positives, false negatives, review coverage, policy-decision latency, enforcement failures, effect duplication, and the time from a policy or detector update to complete fleet convergence. Red-team results complement these contracts; they do not replace them. The next chapter develops that adversarial evaluation discipline in detail (Chapter 58).

Runtime safety as external control

Runtime safety does not require believing that one guard model is unbreakable. It requires knowing which claim each component can support. Detectors estimate; policy decides; enforcement mediates every effect; containment bounds a miss; and receipts make the result testable. Small moderation models remain valuable because they can run on every request and change independently of the generator. Their value is strongest when an uncertain classification feeds an explicit decision and weakest when it is treated as a security boundary by itself.

This is the serving-time form of external control. Training shapes what the model tends to propose. Runtime architecture determines what those proposals are allowed to disclose or do.

Further reading