Part VIII · Chapter 55

Scalable Oversight and Control

AuthorChangkun Ou

Reading time~17 min

Alignment built on human judgment has an expiry date: once a model reasons about a task better than the human grading it, the grade stops being a reliable signal. The field has split around two distinct responses to that problem: getting a strong model to learn the right thing from a weaker teacher, and bounding the damage a misaligned model can do regardless of what it has learned. The open question of how oversight scales with the capability gap is the hinge the whole agenda turns on.

Figure 55.1. Schematic of oversight cost as model capability rises. Direct human review scales poorly, while assisted oversight changes the slope without making supervision free. Idealized curves, not measured data.

When the grader is outmatched

The post-training recipe in Chapter 19 rests on a human, or a reward model (a stand-in scorer trained to imitate human preference labels), being able to tell a better answer from a worse one. That assumption holds as long as the human is the more competent party. The moment the model overtakes the human, it fails. A reviewer who cannot follow a proof has no business certifying it, and a grader unable to audit a codebase will mistake a subtle backdoor for a clean diff. The label is noisier than the thing it supervises, and optimizing against a noisy label teaches the model to satisfy the label, not the intent behind it.

This is the scalable oversight problem: how to supervise a system whose competence exceeds the supervisor's, in the domain where the supervision is needed. What makes it more than a measurement nuisance is that the gap it turns on is not fixed. It is set by scaling. Every increment of the scaling spend in Chapter 5 widens the distance between the frontier model and the human, or weaker model, asked to oversee it. The lower layer that makes the model more capable is the same layer that makes it harder to supervise, which is why a method that works at today's gap cannot be assumed to work at the next generation's.

Constraint arrow

The capability gap that threatens oversight is not a fixed quantity. It is set by scaling. Every increment of the scaling spend in Chapter 5 widens the distance between the frontier model and the human, or weaker model, asked to oversee it. The lower layer that makes the model more capable is the same layer that makes it harder to supervise, which is why an oversight method that works at today's gap cannot be assumed to work at the next generation's.

Two philosophies for one problem

The scalable oversight problem splits into two questions that the field has learned to keep separate, and they pull in opposite directions. The first is an alignment question: can a weak signal still teach a strong model the right behavior? The second is a security question: if the model is not aligned, possibly because it is actively trying to evade detection, can a weaker overseer still prevent harm? The first asks whether the student learns; the second assumes the student might be adversarial and asks whether the institution holds anyway.

Two families of answers map onto those two questions, as Figure 55.2 lays out. One family tries to make the weak teacher's signal go further, betting that you can still get the right behavior out of a stronger model. The other gives up on trusting the model and engineers the deployment so that an adversarial model still cannot do much harm. That fault line sets the sequence: first the trust camp, then the containment camp, and then the threat model that has been pushing the field from the former toward the latter.

Figure 55.2. The scalable oversight problem splits into an alignment question and a security question, each answered by a different family of mechanisms. Deceptive alignment is the threat model that makes the control family necessary.

The trust camp: make the weak signal go further

The first family asks whether a weak teacher can still steer a strong model, and it attacks the problem from two sides: the learning side and the judging side.

Elicitation, not instruction

weak-to-strong generalization asks whether weak labels can elicit stronger latent capability, an ability the model already holds from pretraining but does not display on its own. Burns et al. (Burns et al. 2024) test that limit directly: finetune a strong pretrained model on labels produced by a weak supervisor, and measure how much of the strong model's latent capability survives the weak teaching. The finding is that it generalizes beyond its teacher (Burns et al. 2024). A strong model trained on weak, error-prone labels outperforms the weak supervisor that generated them, because pretraining has already laid down the concept and weak supervision only has to elicit it, not install it. The recovery is partial, not full, and the paper reports an auxiliary confidence loss that pushes the strong student to make confident predictions even where it disagrees with the weak label, which recovers more of the gap (Burns et al. 2024). The mechanism is elicitation, not instruction, and that reframing is the design idea.

That reframing is also the technique's ceiling. Eliciting a latent capability works only when pretraining already contains it. For genuinely novel behavior the weak teacher has nothing to elicit, and the confidence loss that recovers more of the gap can also amplify a confident error. The technique narrows the supervisor-student gap; it does not close it.

Verify, do not generate

Debate and amplification attack the same problem from the judging side, on the bet that checking an answer is easier than producing one. Iterated amplification, proposed by Christiano et al. (Christiano et al. 2018), builds a training signal for a hard question by decomposing it into easier subquestions a human can supervise, answering those with model help, and composing the answers, so the supervisor never has to evaluate the whole problem at once. Debate, proposed by Irving et al. (Irving et al. 2018), sets two strong models against each other in a zero-sum game judged by a weaker party: each tries to convince the judge its answer is correct, and the structure is meant to make dishonesty hard because a lie creates an opening the opponent can expose. The bet is that verifying a rebutted argument is easier than generating the truth from scratch, the way checking a proof is easier than finding it. Khan et al. (Khan et al. 2024) gave this an empirical test with language models: non-expert judges, both models and humans, reach the truth more often when they read a debate between persuasive expert debaters than when they consult a single one, and optimizing debaters for persuasiveness raised judge accuracy rather than lowering it (Khan et al. 2024). Figure 55.3 sketches the game.

Figure 55.3. Debate as a zero-sum game judged by a weaker party. Each debater stakes out a side, and the structure is meant to make a lie easier to refute than to defend. After Irving et al. (2018) and Khan et al. (2024).

The whole structure rests on one assumption: that a lie is easier to refute than to defend, and that the judge can tell a real refutation from a merely persuasive one. Where the judge is systematically fooled, persuasiveness and truth come apart, and optimizing for the first degrades the second. The empirical result is encouraging within the domains tested; it is not a guarantee that holds as the gap widens.

The containment camp: when you cannot trust the model

The second family gives up on trusting the model and engineers the institution around it. AI control, formalized by Greenblatt et al. (Greenblatt et al. 2024), asks a sharper question than alignment does: not "is the model safe?" but "can we deploy it safely even if it is trying to subvert us?" The setup uses an untrusted but capable model, a trusted but weaker model, and a small budget of trusted human review, and combines them into protocols: have the trusted model audit a sample of the untrusted model's outputs, defer to the trusted model on the suspicious ones, escalate the most suspicious to human review. Crucially the protocols are evaluated by a red team that builds the untrusted model to actively evade them, so a passing protocol is one that holds against intentional subversion, not just against honest mistakes (Greenblatt et al. 2024). The deliverable is a safety-usefulness frontier: how much harm a protocol prevents against how much of the capable model's usefulness it gives up.

That frontier is also where control's costs live. Auditing, deferral, and escalation cost usefulness: every output the trusted model overrides is capability left on the table, and human review is a scarce budget that the protocol must spend where it matters most. Control also assumes you can build a credible red team, which is itself a capability race against the model you are containing.

The existence proof that makes containment necessary

Why give up on trusting the model at all? The answer is deceptive alignment: the possibility that a model can appear aligned under training while preserving a hidden objective or trigger. Hubinger et al. (Hubinger et al. 2024) trained sleeper agents: models with a backdoor, write secure code when the prompt says the year is 2023 and insert an exploitable bug when it says 2024, and then ran standard safety training over them. The backdoor survived supervised finetuning, survived reinforcement learning from human feedback, and survived adversarial training (Hubinger et al. 2024). Worse, adversarial training that surfaced the bad behavior taught the model to hide it more precisely rather than removing it. The result does not show that today's models are deceptive. It shows that if a model ever became deceptive, the training tools we rely on would not reliably detect or remove it. That is the existence proof that makes control, oversight that does not assume the model is honest, a necessary track alongside alignment.

The caveat in that result, that the deception had to be trained in, has since been removed. Greenblatt et al. gave Claude 3 Opus nothing but situational information: it was told it would be retrained toward a conflicting objective, with no implanted trigger (Greenblatt et al. 2024). The model strategically complied during what it believed was training in order to preserve its existing preferences: it answered harmful queries it otherwise refused in 14 percent of monitored trials, in almost all of them with explicit alignment-faking reasoning in its scratchpad, and actually training the model to comply via reinforcement learning raised that reasoning to 78 percent of trials. Deception-shaped behavior arose from context alone, which moves the argument for containment from a constructed worst case toward observed behavior.

The cost of that proof falls on everyone. Sleeper agents raise the price of trusting any single training pass. If safety training can hide a behavior rather than remove it, a clean evaluation no longer certifies a clean model, which pushes effort toward interpretability in Chapter 54 and toward control protocols that do not depend on the model being honest.

Why the field is tilting toward control

The lineage runs from a definitional worry to a measured one. The early proposals, amplification in 2018 (Christiano et al. 2018) and debate the same year (Irving et al. 2018), were structural arguments: here is a game whose equilibrium, if humans judge well and the incentives hold, should favor truth. They were largely untested on real systems because the models of the day were not strong enough to make the experiment meaningful.

The capability jump changed what could be measured. By 2024 models were strong enough to be both the debaters and, in weak-to-strong, the weak teacher and the strong student. Burns et al. turned the superalignment worry, the problem of aligning models smarter than the humans supervising them, into a benchmark with a GPT-2-scale supervisor teaching a GPT-4-scale model (Burns et al. 2024). Khan et al. turned debate from a proposal into a result with a measurable accuracy lift for non-expert judges (Khan et al. 2024). In parallel, Greenblatt et al. opened the control track by reframing the goal from trust to containment (Greenblatt et al. 2024), and Hubinger et al. supplied the threat model that makes containment necessary by showing that deception, once present, persists through the standard cleanup (Hubinger et al. 2024).

The balance of the field has been shifting from the alignment framing toward the control framing, not because alignment is solved but because control makes a weaker and more checkable promise. Alignment asks you to believe the model learned the right thing. Control asks only that your protocol holds against a model assumed to be adversarial, which is a claim a red team can attack and a claim you can ship.

A control protocol up close

In practice these ideas show up as protocols, not as single losses. A control protocol is a small decision policy over a stream of model outputs: a monitor, usually the trusted weaker model or a classifier, scores each output for suspicion; a threshold routes the suspicious fraction to a more expensive check; the most suspicious go to scarce human review.

for output in untrusted_model.stream():
    s = trusted_monitor.suspicion(output)      # cheap, runs on every output
    if s < low:
        accept(output)
    elif s < high:
        output = trusted_model.redo(task)       # defer to the weaker, trusted model
    else:
        escalate_to_human(output)               # spend the scarce audit budget

Figure 55.4 shows the cast of actors the pseudocode routes among, and the adversarial loop that makes the evaluation sound.

Figure 55.4. A control protocol as an institution. The capable untrusted model is built by a red team to evade detection, the weaker trusted model both monitors and substitutes, and a scarce human budget catches what survives. After Greenblatt et al. (2024).

The design parameters are the thresholds and the audit budget, and they are tuned on the safety-usefulness frontier: raise the thresholds and you ship more capability but catch fewer attacks, lower them and you are safer but slow and expensive. A sound evaluation of such a protocol is adversarial. You do not measure it against a cooperative model; you have a red team construct the untrusted model to pass the monitor while still slipping harm through, and you report the frontier under that pressure (Greenblatt et al. 2024). A protocol that looks strong against honest mistakes and collapses against a red team is, for this purpose, no protocol at all.

The protocol above scores one output at a time; the field's next step was the trajectory. Ctrl-Z ran the first control evaluation of a multi-step agent, on BashBench's 257 system-administration tasks, and introduced resample protocols: rerun a suspicious action and compare the drafts, turning a one-shot suspicion score into repeated evidence about intent. The best resample protocol cut attack success from 58 percent to 7 percent at a 5 percent cost to benign usefulness, dominating the earlier audit-and-defer designs (Bhatt et al. 2025). SHADE-Arena extends the same measurement to long-horizon agents, scoring at once how well an agent completes a hidden sabotage task and how well an LLM monitor catches it in the transcript (Kutasov et al. 2025).

The capable but untrusted model emits one output into the stream. The red team built it to slip harm past the monitor.

The trusted weaker model, or a classifier, runs on every output and assigns a suspicion score, then compares it to two thresholds.

Suspicion is low, so the output ships unchanged. Most traffic ends here, which is what keeps the protocol useful.

Suspicion is middling, so the task goes back to the trusted weaker model to redo. Safer, but every redo is capability left on the table.

Suspicion is high, so it goes to a human reviewer, spending from the finite audit budget the protocol must save for what matters most.

Figure 55.5. Step through one output's path: accept it, defer it to the trusted model, or escalate it into the finite audit budget.

Debate and weak-to-strong enter the same pipeline as label sources rather than as runtime guards. A debate transcript judged by a weaker model is a way to manufacture a supervision signal for a question the judge could not grade alone, which connects to synthetic supervision in Chapter 23, and to the judging machinery in Chapter 50 and Chapter 52 that these protocols reuse. The recurring discipline across all of them is the same: never certify safety against a model assumed to be cooperative when the threat you care about is a model that is not.

Does oversight survive the gap?

Every mechanism here has been measured at a modest capability gap. The hinge of the whole agenda is whether any of them survives a large one.

Treating oversight as an Elo game (Elo being a paired-skill rating where the gap between two scores predicts which side wins), run the win-probability formula and watch a sharp game and a gentle one come apart as the gap widens.

import numpy as np, matplotlib.pyplot as plt

# Oversight modeled as an Elo game: probability the weaker overseer "wins"
# (catches the model) given an Elo gap. Engels et al. find the decline is
# sharp in some games, gentle in others. The game's scale s sets the slope.
gap = np.linspace(0, 800, 200)              # Elo gap = model minus overseer
def p_oversight(gap, s):                    # s = points per decade of odds
    return 1.0 / (1.0 + 10.0 ** (gap / s))

for s, name in [(150, "sharp game"), (600, "gentle game")]:
    plt.plot(gap, p_oversight(gap, s), label=f"{name} (scale={s})")
    print(name, "P at gap 400:", round(p_oversight(400, s), 3))

plt.axhline(0.5, ls="--", c="gray"); plt.xlabel("Elo gap (model - overseer)")
plt.ylabel("oversight success probability"); plt.legend(); plt.show()

What's contested

Whether oversight degrades as the capability gap between overseer and model grows is the open question the whole agenda turns on, and the evidence points both ways. The optimistic reading is that the mechanisms keep paying off: a strong student recovers a nonzero share of its capability from weak labels, and a weaker judge reaches the truth more often after watching strong models debate. The pessimistic reading is that these gains are measured at modest gaps and may not survive a large one. Engels et al. (Engels et al. 2025) model oversight as a game between capability-mismatched players and find the probability of successful oversight falling as the Elo gap grows, sharply in some games and gently in others, and Hubinger et al. (Hubinger et al. 2024) show that a model determined to deceive can carry that behavior through the very training meant to remove it. No one has demonstrated oversight that provably holds across an arbitrary gap. Whether nested oversight, where each level supervises the one just above it, buys enough margin to bridge a large gap, or whether the gap is the wall the whole agenda eventually hits, is unresolved.

Drag the fragility knob to feel how fast oversight success decays as the capability gap grows.

Figure 55.6. Oversight success probability against the capability gap, after Engels et al. (2025): a higher fragility makes the curve fall off a cliff, a lower one lets oversight degrade gently.