Adversarial Robustness and Red-Teaming

Alignment is a trained disposition, not a guarantee. A model that refuses a harmful request in the average case still holds, somewhere in its weights, the capability the refusal is meant to suppress, and an adversary with access to the input can search for the phrasing that recovers it. That search, and the effort to defeat it, now has a stable shape. Four families of jailbreak attacks, prompts designed to bypass refusal, each exploit a different surface. red-teaming has become a measurement discipline rather than a one-time audit. And because every known defense is partial, adversarial robustness turns out to be a property of the system, not of the model alone.

The boundary with the two neighboring chapters matters. Chapter 57 adds a filter around the model that inspects inputs and outputs; Chapter 56 bounds what a compromised agent can reach by scoping its authority. The problem sits between them, on the manipulation itself: how an attacker turns a refusal into compliance, how a defender measures the rate at which that succeeds, and whether the rate can be driven to zero at all.

The problem: a refusal is a soft boundary

Post-training installs refusal by example (Chapter 19): the model is shown harmful requests paired with declines and learns to reproduce the pattern. The result is a decision boundary in a space the operator cannot enumerate. Refusal holds densely over the prompts seen in training and thins out in the corners no one wrote down. An attack is a procedure for finding those corners, and the unsettling discovery of the period is that the corners are not rare accidents but a reliably reachable surface, often by automated search.

Four families of attack have stabilized, and they differ in what part of the model they exploit.

The first family is optimization-based. The Greedy Coordinate Gradient (GCG) attack treats jailbreaking as a search problem: append a suffix of tokens to a harmful request and optimize the suffix by gradient, the signal that says which token change pushes the model toward the target answer, over a white-box model whose weights are fully visible, to maximize the probability that the model begins its answer with an affirmative phrase (Zou et al. 2023). The suffixes look like noise, they are found by machine rather than by a person, and the central result is that they transfer: a suffix tuned against open weights often jailbreaks closed models it never saw. Optimization turned jailbreaking from craft into a reproducible pipeline.

The second family exploits scale in the context window rather than the token search. Many-shot jailbreaking fills the prompt with a long fabricated dialogue in which an assistant answers a sequence of escalating harmful questions, then asks the real one (Anil et al. 2024). The model, completing a pattern, extends the fabricated assistant's compliance to the final question. The attack's effectiveness grows with the number of shots in a roughly power-law fashion, so the same long-context capability that makes a model useful (Chapter 35) widens the surface this attack runs on.

The third family is conversational. Rather than break the boundary in one shot, a multi-turn attack walks the model across it. A persona attack asks the model to role-play a character not bound by the policy; a crescendo attack opens with a benign request and escalates over several turns, each step a small increment from an answer the model already gave, so no single turn looks like a violation (Russinovich et al. 2024). These attacks exploit that the model conditions on its own prior outputs, and that a request is judged against the visible turn rather than the trajectory.

The fourth family does not come through the user at all. Indirect prompt injection plants instructions in content the agent will read as data: a web page, a document, a tool's output (Greshake et al. 2023). When the agent ingests that content, it cannot reliably tell the planted instruction from the operator's. This is the family that breaks agents specifically, because an agent's reason for existing is to read untrusted external content and act on it. It is the hinge between adversarial robustness and Chapter 56, and the constraint arrow below names why no model-level fix closes it.

Red-teaming as measurement, not audit

If attacks are a reliably reachable surface, then a one-time security review is the wrong instrument. What a deployer needs is a continuous estimate of how often an adversary succeeds, recomputed as both the model and the attacks change. Red-teaming is that estimate, and treating it as a measurement discipline, with a metric, a dataset, and a reproducible harness, is what separates it from anecdote.

The metric is attack success rate: over a fixed set of harmful behaviors and a fixed attack method, the fraction of attempts that elicit a genuinely harmful completion. Whether the number can be trusted lives in the judge that decides what counts as harmful, which is itself a model or a rubric and carries the same construct-validity worries as any judge (Chapter 50). A standardized harness fixes the behaviors, the attacks, and the judge so that two models or two defenses can be compared on the same footing; HarmBench is one such framework, pairing a behavior set with a battery of attacks and an automated classifier (Mazeika et al. 2024). The harmful-behaviors dataset that the GCG work introduced, often called AdvBench, became a common substrate underneath many of these evaluations (Zou et al. 2023).

Generating the attacks is itself automatable, and this is where red-teaming scales past a human team. A language model can be pointed at a target and asked to generate prompts that make it misbehave, then trained on which of its prompts succeeded, turning attack discovery into its own optimization loop (Perez et al. 2022). Automated red-teaming widens coverage far beyond what a human team can hand-write, at the cost that it finds the attacks its generator is shaped to find and can miss whole families the generator never models. Human red teams remain the source of genuinely novel attack classes; the automation amortizes them across every model release.

Defenses, and why each is partial

Defenses divide by where they intervene, and reading them as a defense-in-depth stack rather than a search for the one fix is the realistic posture.

Training-time hardening adds adversarial examples to the post-training mix so the refusal boundary covers more corners. It raises the floor against known attack styles and generalizes poorly to optimized suffixes found after the model froze, because the attacker optimizes against the deployed boundary while the defender trained against last quarter's.

Inference-time classifiers, the subject of Chapter 57, place a separate model on the input and the output to catch attacks the target itself missed. They are independent of the target's alignment, which is their strength, and they add a latency and cost tax on every call while missing phrasings their own training never covered.

Representation-level defenses intervene inside the model rather than around it. Circuit breakers reach into the residual stream, the running vector of activations that carries information through the model's layers, and reroute the internal representations that precede a harmful completion, so that when an attack steers the model toward producing harm, the rerouted representation derails the generation into incoherence rather than compliance (Zou et al. 2024). The appeal is that the defense attaches to the harmful internal state rather than to surface phrasing, which is what optimized and many-shot attacks vary. Whether it generalizes durably or is re-broken by attacks that find new internal routes is exactly the open question the next box names, and it ties this work to the interpretability program in Chapter 54, which supplies the picture of internal features the defense acts on.

System-prompt hardening, instructing the model to distrust embedded instructions and to treat tool content as data, is the cheapest layer and the weakest in isolation: it is itself just text in the context window, subject to the same override the attacks demonstrate.

What robustness costs

Every defense here changes more than one operating axis. The capability cost is over-refusal: a model hardened against attacks declines benign requests that resemble them, and the same boundary that blocks a jailbreak blocks a security researcher's legitimate question, so robustness and helpfulness are tuned against each other rather than independently maximized. The efficiency cost is the classifier tax of Chapter 57 and the training cost of adversarial and representation-level methods, paid on every request or every release. Trust is the point of the whole exercise, and the credible form of it is a published attack success rate against a named harness rather than a claim of safety: robustness that cannot be measured the way HarmBench measures it (Mazeika et al. 2024) is an assertion, not a property. The discipline's contribution to Chapter 55 is precisely this, a number that says how far the deployed boundary can be pushed and by whom, recomputed every time the model or the attack changes.

Further reading