Adversarial Robustness and Red-Teaming

Adversarial robustness asks how a deployed system behaves when someone deliberately searches for a failure. A jailbreak attempts to elicit policy-violating model output. Prompt injection attempts to make untrusted text control an application. A harmful side effect additionally requires the application to authorize and execute the model's proposal. These failures are related, but they are not interchangeable.

This is the task of red-teaming. Red teaming searches for them under a stated threat model. It does not certify their absence. The result is evidence about one system revision, policy, attacker, budget, and evaluator, not a timeless property of a model family. adversarial robustness therefore belongs to the whole deployed system: the model, prompts, tools, guards, authorization rules, and release process all affect the outcome.

Begin with the failure, not the prompt

A refusal string is observable, but it is not the safety objective. An answer can avoid the word “no” and still provide no harmful help. It can also begin with a warning and then provide prohibited operational detail. For an agent, unsafe text is still different from an external effect. The system may generate a bad proposal that a deterministic gate rejects before anything changes.

This separation determines where a defense belongs. Chapter 22 changes the model's default behavior. Chapter 57 detects and gates request-time events. Chapter 56 decides which effects are permitted. A model-level evaluation can measure the first outcome, but it cannot establish the last three without running the relevant application path.

Write the threat model before running attacks

An attack success rate is uninterpretable until the evaluator fixes the attacker's powers (Carlini et al. 2019). JailbreakBench makes this explicit by versioning behaviors, prompts, chat templates, scoring functions, and the threat model (Chao et al. 2024). A useful evaluation record answers each of these questions:

Comparisons require the same contract. Results from different policies, budgets, sampling settings, or judges should not be placed in one ranking without qualification. Even within one contract, a vendor API may change underneath an evaluation; record the observed model revision and date rather than relying on a product name alone.

Attack methods expose different surfaces

The modern literature made attack discovery reproducible rather than eliminating the role of human testing. Perez et al. used one language model to generate test cases for another in 2022 (Perez et al. 2022). Zou et al. introduced Greedy Coordinate Gradient (GCG) in 2023, showing that optimized adversarial suffixes could transfer from open-weight models to several black-box systems evaluated at the time (Zou et al. 2023). Greshake et al. separately showed that instructions planted in retrieved data could influence LLM-integrated applications (Greshake et al. 2023). These works address different goals and access models.

The following mechanisms are useful axes, not an exhaustive taxonomy. They include jailbreak attacks, prompts designed to bypass refusal or other learned policy behavior, as well as prompt injections. An attacker can combine them in one campaign.

Manual and semantic attacks use role-play, reframing, translation, indirection, or other meaning-preserving changes. They can be found by people or generated by another model. The important variable is not whether a prompt looks clever; it is whether the method produces policy-violating, useful output within the stated budget.

Optimization-based attacks search for tokens that raise the likelihood of a target continuation. GCG uses gradients from a white-box source model to propose suffix-token changes (Zou et al. 2023). Its reported transfer to named black-box models established that some optimized prompts cross model boundaries, not that every suffix transfers to every model. Black-box methods instead use target queries, response scores, or a separate attack model; their query budget is part of the result.

Many-shot attacks fill a long context with fabricated demonstrations in which the assistant complies, then append the behavior under test. Anthropic reported that attack effectiveness increased approximately as a power law with the number of demonstrations in the evaluated models and tasks (Anil et al. 2024). That empirical shape is not a universal law, but it shows why a longer context window changes the attack budget.

Multi-turn attacks use conversation state. Crescendo begins with a benign topic and incrementally moves toward a prohibited request while reusing the model's earlier answers (Russinovich et al. 2025). The paper reports success against its evaluated systems. The general lesson is narrower: a guard that assesses only the latest message can miss intent that becomes clear across the trajectory.

Indirect prompt injection places instructions in a web page, document, email, or tool result that an application later treats as data (Greshake et al. 2023). It is not merely another way to obtain a harmful completion. Its goal is to redirect an application's control flow, data flow, or tool use. A behavioral defense may reduce measured injection success, but only an external enforcement point can decide whether the resulting effect is authorized.

Measure attempts, not anecdotes

Attack success rate (ASR) is a useful summary if its unit of analysis is explicit. Let $b$ be a behavior under test, $a$ an attack procedure, $r$ a generation seed or trial index, and $y_{b,a,r}$ the target system's response. Let $J_v$ be the policy-aware judge at revision $v$. Define the binary outcome

$$Y_{b,a,r} = \mathbf{1}[J_v(b,y_{b,a,r})=\mathrm{success}],$$

where $Y_{b,a,r}=1$ means that response $y_{b,a,r}$ satisfies the documented success criterion for behavior $b$; otherwise $Y_{b,a,r}=0$. The indicator $\mathbf{1}[\cdot]$ maps a true statement to one and a false statement to zero.

For an evaluation set $\mathcal{T}$ of behavior, attack, and trial triples, define $N=|\mathcal{T}|$ and

$$\widehat{\operatorname{ASR}} = \frac{1}{N} \sum_{(b,a,r)\in\mathcal{T}}Y_{b,a,r}.$$

Here $\widehat{\operatorname{ASR}}$ is the observed attack success rate, $N$ is the number of evaluated triples, and the sum counts successful attempts. This point estimate is not a probability that the system is safe. It describes the sampled behaviors, attacks, seeds, judge, and system revision.

Report the aggregation rule. “Per-attempt ASR” gives every trial equal weight. “Any-success ASR” marks a behavior as broken if any restart succeeds. These answer different questions and diverge as the restart budget grows. Also report results per risk category, along with the severity and actionability of successful outputs; one average can hide a rare but consequential failure.

Validate the judge

Bypassing a refusal is not sufficient. StrongREJECT found that common evaluators could overstate jailbreak effectiveness when a model stopped refusing but produced an empty or incoherent answer (Souly et al. 2024). A useful judge scores harmful usefulness: did the answer actually provide the prohibited capability with enough specificity to matter?

Automated judging makes large sweeps affordable, but Chapter 50 still applies. Draw a stratified human-review sample from successes, failures, high-severity categories, and borderline scores. Report judge agreement and estimate the false positive and false negative rates against policy-trained reviewers. Keep the judge revision fixed within a comparison. If the judge changes, rescore both candidates or treat the runs as different experiments.

Report uncertainty and dependence

Stochastic decoding means the same behavior and attack can yield different answers, so run repeated samples when deployment uses sampling. Publish the sample size and a confidence interval, not only a percentage. Multiple attacks and restarts against one behavior are not independent evidence. When estimating uncertainty over behavior coverage, resample by behavior rather than treating every correlated retry as a new independent case.

The interval method should match the design: a binomial interval may fit one independent outcome per behavior, while a cluster bootstrap can preserve repeated attempts within each behavior. State the method and grouping. Zero observed successes does not prove zero risk; it only places a sample-dependent upper bound under the evaluation assumptions.

Preserve utility and cost

A lower attack success rate is not enough if the defense refuses most benign requests, damages ordinary task quality, or makes the service impractical. Pair harmful behaviors with benign contrasts that share vocabulary and context. XSTest is one such contrast set for exaggerated safety behavior (Röttger et al. 2024). Report benign refusal, ordinary task quality, and safe-completion usefulness alongside ASR.

Attack cost also changes the meaning of a result. Record queries, tokens, wall-clock time, restarts, and any external compute used by the attacker. Record defense overhead in latency, token use, accelerator time, and monetary cost. A defense that converts a one-query attack into a thousand-query attack may be useful even if it does not make ASR exactly zero, but that claim requires a rate-limit and identity model that makes the budget real.

Build a red-team program, not one benchmark run

HarmBench standardizes behaviors, attacks, target models, and an automated classifier so methods can be compared on common ground (Mazeika et al. 2024). JailbreakBench adds an artifact repository and explicit threat-model components (Chao et al. 2024). These benchmarks make results more reproducible; neither covers every deployment policy, language, modality, tool, or attacker.

A durable program separates four sources of evidence:

1. A development suite gives engineers rapid feedback on known failures.
2. A held-out suite contains unseen prompts and an attack-family holdout. Do not tune on the release suite.
3. An adaptive suite gives a defense-aware attacker the deployed prompt, guard behavior, or enough queries to infer them. Simple adaptive changes have broken defenses that looked strong under static evaluation (Andriushchenko et al. 2025).
4. A human red team explores product-specific workflows, social context, novel combinations, and harms not represented by the automated generators.

Keep part of the held-out suite private and rotate it to reduce direct overfitting. Track lineage so training data, attack-generator prompts, benchmark items, and release cases do not silently overlap. Contamination does not make a case useless as a regression, but it does weaken a claim about generalization. Replay every confirmed failure in the development suite, and reserve new behaviors or whole mechanisms for later held-out runs.

NIST's Generative AI Profile recommends adversarial role-playing, GAI red-teaming, or chaos testing to identify unforeseen failure modes (Autio et al. 2024). In practice, that means running this loop before release, after changes to the model or surrounding system, and after incidents reveal a new path:

Match the defense to the failure stage

No single defense covers every row in the opening table. Separate learned model defenses, learned runtime detectors, and deterministic effect controls. For each intervention, state both the measured improvement and the stronger claim that the evidence does not establish.

Safety tuning and adversarial training add known failure cases and policy contrasts to post-training. They can improve both direct refusal and safe completion, but their generalization depends on the training distribution, policy, model, and attack budget. An adaptive attacker should be evaluated after the weights and defense are fixed.

Instruction hierarchy trains the model to prioritize privileged instructions and treat lower-authority or untrusted content appropriately. It is a learned defense, as are prompt transformations that mark data provenance. Both can lower measured prompt-injection success, but neither is an authorization decision. The training objectives and empirical limits are developed in Chapter 22.

Input and output classifiers can catch failures that the target model misses. A separate service is operationally distinct, not automatically statistically independent: the target and guard may share training data, policy ambiguities, architecture, or blind spots. Measure their residual-error correlation. Input filters can block or transform a request; output filters can gate release or tool proposals, but cannot undo text already streamed or a side effect already committed (Chapter 57).

Representation rerouting, presented as circuit breakers, trains internal representations associated with harmful continuations away from their normal trajectory. Zou et al. reported lower ASR across the attacks and model types they evaluated while largely preserving measured utility (Zou et al. 2024). That is empirical generalization to a reported suite, not proof of a unique harmful circuit or a guarantee against an adaptive attacker. Defense-aware evaluations have subsequently shown why the threat model must be restated for every robustness claim (Andriushchenko et al. 2025).

Deterministic effect controls enforce typed tool schemas, authorization, recipient and destination rules, least privilege, approval binding, secret custody, and sandbox or egress constraints. These controls do not make model output safe. They create a security boundary for the narrow properties that trusted code checks, so a successful jailbreak or injection does not automatically become an external compromise (Chapter 56).

Turn results into a release rule

A release gate should be written before the run. For each risk category $c$, one possible rule is

$$\operatorname{UCB}_{1-\alpha}(\operatorname{ASR}_c) \le \tau_c,$$

where $\operatorname{ASR}_c$ is the attack success rate for category $c$, $\operatorname{UCB}_{1-\alpha}$ is its upper confidence bound at confidence level $1-\alpha$, $\alpha$ is the chosen error probability, and $\tau_c$ is the documented release threshold for that category. A severe category may receive a tighter threshold and a larger sample than a lower-impact one.

Safety alone is not the whole gate. Require

$$\operatorname{LCB}_{1-\alpha}(U_{\mathrm{benign}}) \ge u_{\min},$$

where $U_{\mathrm{benign}}$ is the chosen benign-utility measure, $\operatorname{LCB}_{1-\alpha}$ is its lower confidence bound, and $u_{\min}$ is the benign-utility floor. Thus a lower ASR obtained by refusing everything still fails the release gate. Thresholds are product and risk decisions, not constants supplied by the benchmark.

If evidence misses a threshold, fix the system, narrow its deployment, add an independent control, or document an authorized exception with an owner and expiry. Do not discard unfavorable runs or change the judge after seeing the result. Chapter 53 explains how to keep release evidence separate from development feedback.

Keep a reproducible evaluation record

Store one record that connects the system, threat model, attempts, judgments, utility results, and release decision:

evaluation_id
system_revision
policy_revision
defense_revision
behavior_set_revision
attack_revision
threat_model_and_budget
generation_parameters_and_seeds
judge_revision
human_validation_sample
per_behavior_outcomes
benign_utility_outcomes
query_token_latency_costs
confidence_interval_method
release_decision_and_exception
regression_case_ids

Retain attack artifacts under access controls appropriate to their misuse risk. A summary can publish methods and aggregate results without publishing every high-risk prompt. The internal record still needs enough detail for an authorized evaluator to reproduce the claim.

Regression coverage should include more than the latest successful string:

Adversarial robustness is not one score attached to one model. It is a repeatable process: define the failure, state the attacker's powers, run representative and adaptive attacks, validate the judge, quantify uncertainty and utility, enforce a release rule, and preserve every confirmed failure as regression evidence. The result is not a certificate. It is a versioned, auditable claim about how one deployed system resisted one documented adversary.

Further reading