Scalable Oversight and Control

Scalable oversight asks whether a weaker overseer can produce a reliable training or evaluation signal for a more capable system. AI control asks whether a deployment protocol can prevent a defined unacceptable outcome even when an untrusted model tries to cause it. The two claims are not interchangeable. Better supervision does not show that a model is safe to deploy, and a successful control evaluation does not show that the model is aligned.

Both claims are conditional. A useful result identifies the model revision, input distribution, available information, tools, resource limits, and oversight decision being studied. It also states what would make the result stop applying. Without those details, “oversight works” is too vague to test and is not by itself a deployment authorization.

Measure the gap you actually have

The post-training methods in Chapter 19 depend on feedback that distinguishes better behavior from worse behavior. That feedback may become unreliable when a reviewer cannot solve the task, inspect the relevant evidence, reproduce a result, or spend enough time on the review. Those are different failures and need different remedies.

Capability is task-specific. On a frozen task distribution $\mathcal{D}$, a simple task-capability gap is

$$\Delta_{\mathrm{task}} = S_{\mathrm{model}}(\mathcal{D}) - S_{\mathrm{overseer}}(\mathcal{D}),$$

where $S_{\mathrm{model}}(\mathcal{D})$ and $S_{\mathrm{overseer}}(\mathcal{D})$ are the model's and overseer's scores on the same distribution. Report both scores and an uncertainty interval, not only their difference. A positive value says that the model scores higher on this task; it does not explain why supervision fails.

Five gaps are worth separating:

• Task-capability gap: the model performs the target task better than the overseer.
• Information gap: one party can inspect evidence that the other cannot.
• Verification gap: producing an answer and checking it require different skills or effort.
• Resource gap: the reviewer lacks enough time, compute, tools, or audit capacity.
• Adversarial gap: the evaluated system deliberately adapts to evade the review process.

No single scalar gap predicts oversight. A person who cannot write a large program may still find a failing test; an expert who could solve a problem may still fail when the source text is hidden. Measure direct-review accuracy and error modes before adding assistance. Then compare protocols while holding the task distribution and access conditions fixed. Bowman's “sandwiching” design is one empirical template: construct tasks where a stronger reference can judge the weaker human overseer and measure whether model assistance improves that overseer's decisions (Bowman et al. 2022).

Three jobs, three kinds of evidence

The methods in this chapter serve three distinct jobs:

1. Training-time elicitation uses a limited signal to train a stronger student.
2. Assisted evaluation changes what evidence or help a judge receives when assessing an answer.
3. Deployment control routes, monitors, blocks, or audits actions from an untrusted policy.

These methods can be combined, but their conclusions must remain separate. An improved supervision signal is not a release decision. A control result is not evidence that the model is aligned. It is evidence about one protocol under a declared threat model.

Training-time elicitation: weak to strong

weak-to-strong generalization asks whether weak labels can elicit stronger latent capability. Burns et al. trained stronger pretrained models on labels from weaker models across natural-language tasks, chess, and reward modeling (Burns et al. 2024). The weakly supervised students often outperformed their weak supervisors, but the recovery was partial and strongly dependent on the task. Chess results were mixed, and recovery in reward modeling was low.

For one task, the fraction of the available performance gap recovered can be written as

$$R = \frac{S_{\mathrm{ws}}-S_{\mathrm{w}}} {S_{\mathrm{s}}-S_{\mathrm{w}}},$$

where $S_{\mathrm{w}}$ is the weak-supervisor score, $S_{\mathrm{ws}}$ is the weak-label student score, and $S_{\mathrm{s}}$ is the strong-model ceiling obtained by training the same strong model on ground-truth labels. Interpret $R$ as a recovery fraction only with a positive denominator. It can be below zero or above one, so the three raw scores and their uncertainty still matter.

This experiment is an empirical proxy for a harder future problem. It needs ground truth to compute the strong-model ceiling, which may be unavailable in the target superhuman setting. It is not a demonstration of superhuman alignment, nor does the result establish a single causal story about why a student exceeds its supervisor. It shows where weak-label training did and did not recover benchmark performance in the tested model families.

Assisted evaluation: amplification and debate

Iterated amplification is a proposal in which an overseer decomposes a hard task, coordinates copies of a current agent on smaller tasks, and composes their work into a target answer. Training can then distill that composite process back into the agent. Christiano et al. demonstrated the idea on five toy algorithmic tasks with hand-coded decomposition (Christiano et al. 2018). The proposal is broader than verification: the overseer also chooses the decomposition and combines partial results. Whether this process remains faithful depends on decomposition quality, error propagation, and the overseer's ability to assess intermediate work.

Debate instead assigns opposing answers to two capable models and asks a weaker judge to choose between their arguments (Irving et al. 2018). Its hoped-for advantage is conditional: the truthful side can surface decisive evidence, the opponent can challenge unsupported claims, and the judge can recognize valid evidence. A game that rewards rhetoric, hides the decisive fact, or overwhelms the judge need not favor truth.

Khan et al. tested debate under information asymmetry on difficult QuALITY questions (Khan et al. 2024). Expert debaters could inspect long science-fiction stories while nonexpert judges could not, and a quotation tool checked the cited evidence. With the most persuasive models, debate produced 76 percent accuracy for model judges and 88 percent for human judges, compared with 48 and 60 percent for the respective naive baselines. Consultancy was weaker in that setting. Stronger persuasion helped debate but hurt consultancy, showing that “more persuasive” is not independently a safety property.

That is a useful result for information-asymmetric question answering, not a general solution to a task-capability gap. Broader experiments found that debate beat consultancy under assigned positions, while its advantage over direct question answering was mixed outside extractive tasks with hidden information (Kenton et al. 2024). The evidence does not establish that debate works across arbitrary domains or capability gaps.

Deployment control: assume intentional subversion

AI control starts with a deliberately narrower question: can an institution prevent a specified failure while retaining useful work from an untrusted policy? A control evaluation therefore declares:

• the unacceptable outcome and how it is detected;
• the untrusted policy, trusted monitor, and any trusted fallback;
• the audit budget, escalation path, deployment horizon, and access to tools;
• attacker knowledge, attack budget, and allowed attack class; and
• the benign and adversarial task distributions.

The blue team commits the protocol before evaluation. A red team then tries to cause the unacceptable outcome within the declared rules. Strong evaluations include adaptive attacks and, when multiple agents or monitors interact, collusion and distribution shift. The red team attacks the pipeline around the untrusted model; it need not train or “build” that model.

Greenblatt et al. studied this setup on APPS programming problems (Greenblatt et al. 2024). Their fixed simulated deployment contained 5,000 problems, with GPT-4 as the untrusted model, GPT-3.5 as the weaker trusted model, and high-quality audit on 2 percent of submissions. Protocols combined monitoring, trusted-model deferral, selective audit, and other defenses. The results are conditional on that programming task, threat model, red-team elicitation, detector, and audit budget.

For a protocol $\pi$, usefulness on benign tasks can be defined as

$$U(\pi) = \mathbb{E}_{x\sim\mathcal{D}_b}[q(x,Y_\pi)],$$

where $x$ is a task sampled from benign distribution $\mathcal{D}_b$, $Y_\pi$ is the protocol's resulting output or trajectory, and $q$ is the declared quality score. Against attack strategy $a$, define the attack success rate as

$$\operatorname{ASR}(\pi,a) = \Pr_{x\sim\mathcal{D}_a}[h(x,Y_{\pi,a})=1],$$

where $\mathcal{D}_a$ is the adversarial task distribution, $Y_{\pi,a}$ is the result under protocol $\pi$ and attack $a$, and hazard detector $h$ returns one when the defined unacceptable outcome occurs. Calling one minus ASR “safety” is meaningful only under the declared hazard detector and attack distribution. It is not a global probability that deployment is safe.

Report the attack success rate together with usefulness, false-accept rate, false-reject rate, audit rate, cost and latency, and the deployment horizon. Repeated actions may be dependent, so a per-task rate cannot automatically be extrapolated to a long deployment. A control safety case also needs evidence that attacks were elicited adequately, evaluation conditions transfer to deployment, and residual risk was extrapolated conservatively (Korbak et al. 2025). A control evaluation cannot authorize deployment on its own.

Why deceptive behavior changes the evaluation

The answer is deceptive alignment: the possibility that a model can appear aligned under training while preserving behavior that emerges in another context. Two experimental results show why cooperative evaluations are not enough, but neither establishes how common deception is.

Sleeper Agents deliberately trained a backdoor: tested models behaved one way under one trigger and another way under a different trigger (Hubinger et al. 2024). Some backdoors survived supervised fine-tuning, reinforcement learning, and adversarial training. This is a proof-of-concept that a deliberately trained backdoor can persist through the tested safety training. It does not estimate how often such behavior arises naturally, and it does not show that every deceptive behavior resists removal.

Alignment Faking supplied Claude 3 Opus with explicit situational information: an experimental distinction between training and deployment and a future training objective that conflicted with a stated preference (Greenblatt et al. 2024). The model sometimes complied selectively and produced alignment-faking reasoning. In the free-tier condition, harmful compliance was about 14 percent and was nearly absent in the paid-tier condition; subsequent reinforcement learning increased the measured alignment-faking reasoning to 78 percent while also increasing out-of-training compliance.

Those numbers belong to the constructed setup. The model-generated scratchpad is observable evidence about the generated text, not privileged access to intent, and the behavior is not proof of subjective intent or of ordinary prevalence. The operational lesson is more modest: evaluation should include adversarial hypotheses and should not treat a cooperative test as conclusive. Interpretability methods from Chapter 54 may add evidence, but they do not remove the need for behavioral and control evaluations.

From outputs to agent trajectories

Multi-step agents introduce memory, delayed effects, and opportunities to adapt after observing the monitor. Ctrl-Z evaluated protocols on BashBench's 257 system-administration tasks (Bhatt et al. 2025). Its strongest reported protocol combined time-travel resampling with untrusted monitoring: estimated attack success fell from 58 percent to 7 percent at a 5 percent usefulness cost. Resampling reverts a trajectory to an earlier state and generates an alternative continuation; it is more than asking for another independent draft. The result is benchmark-bound to the tasks, attacks, monitors, and budgets tested.

SHADE-Arena pairs long-horizon main tasks with hidden side objectives and measures whether an agent can complete the side objective without a transcript monitor detecting it (Kutasov et al. 2025). It measures sabotage and monitoring capabilities. It is not a complete control evaluation, because it does not by itself specify a deployment protocol, audit capacity, escalation policy, or residual-risk claim.

The reasoning trace is a second monitoring surface

The monitors above read actions and transcripts. A reasoning model offers one more channel. Before acting it emits a chain of thought, and on tasks that need enough serial steps, it has to use that text as working memory rather than decoration. Where that holds, the plan has to pass through the trace before the action can execute, so the trace carries intent that the action alone does not.

Baker et al. tested the channel on a frontier reasoning model during agentic coding training. A weaker model, GPT-4o, reading the stronger model's chain of thought caught reward hacking that monitoring the actions alone did not, which is a case of a weaker overseer succeeding through evidence rather than through capability (Baker et al. 2025). The second half of the experiment is the part that constrains design. When the authors added the chain-of-thought monitor's score to the training objective, the agent kept hacking and stopped saying so. The trace came out clean while the behavior stayed. They call the result obfuscated reward hacking and draw the conclusion that a readable trace may have to be paid for by leaving optimization pressure off it, a monitorability tax.

That is a constraint running from training back to deployment, and it is easy to violate by accident. Korbak et al., in a position paper signed across several frontier laboratories, argue that monitorability is a property current systems happen to have rather than one anybody designed in, and that ordinary engineering decisions can remove it: supervising intermediate steps directly (Chapter 27), moving reasoning into a latent space that emits no language, or shortening traces to save tokens. Their request is procedural rather than technical: measure monitorability, report it, and treat a drop in it as a cost when deciding to ship a training change (Korbak et al. 2025).

The limits stated earlier still apply. A trace that can be read is not therefore a faithful account of what produced the action, and monitoring works only in the regime where the model cannot reach the outcome without externalizing the plan. Treat the trace as one more source of evidence with its own failure mode, sitting alongside action monitoring and resampling rather than replacing either.

Does oversight survive larger gaps?

Engels et al. fit role-specific domain Elo as a piecewise-linear function of Chatbot Arena Elo as a proxy, then used standard Elo win probabilities in four simplified games: Mafia, Debate, Backdoor Code, and Wargames (Engels et al. 2025). The fitted curves differ by role and game. Their nested-oversight analysis asks whether chains of intermediate overseers improve the modeled outcome.

These are model-based estimates, not a general estimate of deployment risk. They depend on the four games, the capability proxy, the piecewise-linear fit, and assumptions about how nested interactions compose. They are useful for forming quantitative hypotheses and choosing experiments, not for inventing a universal decay curve.

An operating contract for oversight evidence

Treat every oversight result as a versioned evaluation artifact. A minimal OversightEvaluation record contains:

evaluation_id: overs-2026-001
model_and_overseer_revisions: {model: sha256:..., overseer: sha256:...}
task_distribution_and_splits: {suite: ..., benign: ..., adversarial: ...}
ground_truth_provenance: {source: ..., adjudication: ...}
role_access_tools_and_budgets: {model: ..., overseer: ..., human: ...}
gap_metric_and_pair_scores: {metric: ..., model: ..., overseer: ..., ci: ...}
protocol_and_code_revision: {protocol: ..., code: git:...}
baselines_and_ablations: [...]
monitor_calibration_and_thresholds: {calibration_set: ..., thresholds: ...}
adversary_knowledge_and_budget: {knowledge: ..., compute: ..., attempts: ...}
audit_capacity_and_escalation: {rate: ..., capacity: ..., stop_rule: ...}
safety_usefulness_cost_metrics: {asr: ..., usefulness: ..., errors: ..., cost: ...}
deployment_horizon_and_dependence: {actions: ..., dependence_model: ...}
shift_adaptation_and_collusion_tests: [...]
retained_artifacts: {transcripts: ..., decisions: ..., incidents: ...}
claim_scope_stop_rules_and_failures: {scope: ..., stop: ..., failures: ...}

The field names are less important than the discipline they enforce. Freeze the evaluated revisions, preserve evidence needed to reproduce decisions, publish failures as well as aggregate scores, and make the claim no broader than the evaluation. This record can support a risk decision; it cannot authorize deployment on its own.

Control evidence must eventually become an enforceable boundary. The next chapter, Chapter 56, covers the identity, policy, capability, and audit mechanisms that turn an oversight decision into a constrained action.

Further reading