The Capability Horizon and Its Measurement

The frontier is no longer a leaderboard number. It is a moving horizon: a set of task, reliability, cost, and oversight limits that move at different speeds. That makes the frontier not one scalar but a measurement problem. A headline number can still be useful, but only when its measurement contract travels with it: the system tested, the tasks sampled, the resources allowed, the success threshold, and the snapshot date.

This distinction matters because the same evidence can support very different stories. Rising task horizons may be durable progress, or part of the apparent acceleration may be a compute-bought artifact. A model may beat an expert deliverable in a blind comparison while failing to automate the surrounding job. A benchmark may rise quickly because capability improved, because tools changed, or because the instrument became easier to exploit. This chapter separates those claims instead of compressing them into one score.

A horizon is a fitted threshold

The task-completion horizon is not the length of time an agent runs. It uses human completion time as a proxy for task difficulty, then estimates how success changes as tasks become longer. METR's original study focused mainly on self-contained software, machine-learning, and cybersecurity tasks. Its result therefore describes a fixed system and protocol on that task distribution, not the probability of succeeding on every real-world task of the same duration (Kwa et al. 2025).

Let task $i$ have a positive human completion time $d_i$, where $i$ identifies the task. For repeated attempt $r$, let $Y_{ir}=1$ when the evaluated system succeeds and $Y_{ir}=0$ otherwise. A simple version of the fitted model is

$$\Pr(Y=1\mid d)=\sigma(\alpha-\beta\ln d), \qquad \sigma(z)=\frac{1}{1+e^{-z}}, \qquad \beta>0.$$

Here $Y$ is the binary outcome, $d$ is human completion time, $\alpha$ is the fitted intercept, $\beta$ is the fitted slope, $\ln$ is the natural logarithm, and $\sigma$ maps the fitted value to a probability. For a chosen success threshold $q$, the corresponding horizon is

$$h_q=\exp\!\left(\frac{\alpha-\operatorname{logit}(q)}{\beta}\right), \qquad \operatorname{logit}(q)=\ln\!\left(\frac{q}{1-q}\right).$$

$h_q$ means the human-time duration at which the fitted success probability equals $q$. It is an estimate with uncertainty, not an observed task boundary. The model checkpoint, agent scaffold, tool access, task distribution, human-time baseline, repeated attempts, token budget, time budget, scoring rule, and measurement date all belong to its protocol.

The threshold changes the answer

The 50 percent horizon is easy to communicate, but many deployments need much greater reliability. The 80 percent horizon comes from the same fitted curve. Its distance from the 50 percent horizon is

$$\frac{h_{0.5}}{h_{0.8}} =\exp\!\left(\frac{\operatorname{logit}(0.8)}{\beta}\right).$$

In words, the ratio depends on $\beta$, the fitted slope. It is not a universal constant. The original study found ratios around four to six for its fits, while later model estimates varied more widely (Kwa et al. 2025). This is why “five times shorter” can describe one empirical fit but cannot be promoted to a law.

The following calculation uses 18 hours and 3.5 hours only as illustrative midpoints of early-2026 reported ranges. It recovers the slope implied by those two thresholds; it does not refit METR's task-level data.

from math import log

h50 = 18.0
h80 = 3.5
beta = log(4.0) / log(h50 / h80)

print(f"fitted slope: {beta:.3f}")
print(f"50% / 80% horizon ratio: {h50 / h80:.3f}")

A second intuition is sometimes useful but must remain separate. If a task had $n$ independent, homogeneous steps, each succeeding with probability $p$, its all-step success probability would be $p^n$. That toy model shows why long workflows can be fragile. Real agent trajectories do not satisfy its independence or homogeneous steps assumptions, and the horizon studies do not identify step failures as the mechanism. It is an illustration, not a causal explanation. The recovery and compounding problems in Chapter 69 are related operational concerns, not a derivation of the horizon curve.

Trend estimates are conditional

Version 1.1 of the horizon suite made the measurement problem visible. It expanded the benchmark from 170 to 228 tasks and included 31 tasks estimated to take at least eight human hours. Only five of those 31 had measured human baselines; the rest used estimates. In the January 2026 release, Claude Opus 4.5 had a 50 percent horizon of 320 minutes, with an interval from 170 to 729 minutes (METR 2026). That is a wide interval around a roughly five-hour point estimate.

The reported doubling time also changed with the regression window. The hybrid Time Horizon 1.1 series gave 196.5 days over the full history, 130.8 days when fit from 2023, and 88.6 days when fit from 2024. These are different fit windows, not one clock accelerating from seven months to three within a few weeks. Task composition, baseline estimation, model selection, and the short recent window all affect the trend.

By February and March 2026, METR reported a shared internal model with a 50 percent point from 16 to 20 hours and an 80 percent point from three to four hours. The same report warned that the suite cannot reliably measure horizons above 16 hours because only five tasks lie beyond that duration. The shared internal model was not a named public checkpoint, and its runs used 16 to 64 million tokens, above earlier historical budgets (METR 2026). This is evidence that the frontier was pressing against the instrument. It is not a clean continuation of the January public-model series.

Economic tasks measure preference, not jobs

GDPval replaces exam questions with professional deliverables. Its full set has 1,320 tasks from 44 occupations in nine sectors of the US economy. Task writers averaged 14 years of experience. The published launch evaluation used a 220-task open gold subset and asked occupation-matched experts to compare an unlabeled model deliverable with the task writer's deliverable (Patwardhan et al. 2025).

At launch, Claude Opus 4.1 received a better-or-equal judgment in 47.6 percent of graded comparisons. That is wins plus ties, not a plain win rate and not 47.6 percent of jobs automated. OpenAI's 23 April 2026 release later reported GPT-5.5 at 84.9 percent wins plus ties on original GDPval, using xhigh reasoning in a research environment (OpenAI 2026). The later release did not disclose enough detail to assume that every sampling and judging choice remained identical.

The tasks are one-shot, well-specified deliverables. They omit much of a job: deciding what to do, gathering tacit context, negotiating requirements, iterating with a customer, and bearing responsibility for the outcome. Claims of roughly one-hundred-fold lower time or cost refer to pure model inference and API accounting. They exclude human oversight, integration, iteration, and rework. The 84.9 percent figure is therefore a benchmark preference rate, not an occupation-automation rate.

RE-Bench measures a different object. It contains seven self-contained machine-learning research-engineering environments. Its human comparison comprised 61 distinct human experts making 71 eight-hour attempts. At a two-hour total computer-time budget per environment, the best agent configuration achieved about four times the average human normalized score. Humans narrowly led at eight hours and reached about twice the best agent score at 32 hours (Wijk et al. 2025). These are best-of-k results under particular allocations, not necessarily one continuous attempt. A normalized score on seven engineered environments is informative about short-budget search, but it is not end-to-end research automation.

Capability is not productivity

A randomized study makes the transfer gap concrete. From February to June 2025, 16 developers with substantial open-source experience completed 246 tasks in repositories they knew well. With access mainly to Cursor Pro and Claude 3.5 or 3.7 Sonnet, tasks took 19 percent longer. Before the study, developers predicted they would be 24 percent faster; afterward, they believed they had been 20 percent faster (Becker et al. 2025). The result is limited to that sample, those tools, and those tasks. It is still strong evidence that perceived speed and measured speed can diverge.

A February 2026 follow-up produced raw estimates of an 18 percent speedup for returning developers and a four percent speedup for new developers. METR called both signals unreliable: returning participants and tasks were affected by selection, compensation changed, and concurrent agent use made timing a poor proxy (Becker et al. 2026). The update does not overturn the randomized result. It says the original protocol no longer measured newer workflows well. Benchmark capability, task time, self-reported value, and causal productivity are four different quantities. No stable conversion among them is known.

Benchmarks age in different ways

An instrument can lose value through ceiling effects, contamination, protocol drift, or exploitability. These are different failures. A saturated benchmark may still detect regressions in weaker systems; it simply has little headroom at the frontier. The 2026 HLE paper noted that frontier systems had passed more than 90 percent on MMLU, motivating a harder set of 2,500 expert questions over more than 100 subjects (Center for AI Safety et al. 2026).

HLE itself shows why protocol labels matter. On August 7, 2026, the official Scale no-tools leaderboard topped out at 46.44 percent. Separately reported tool-enabled results exceeded 50 percent. No-tools and tool-enabled scores are not points on one clean curve, because retrieval and code execution change the system under test (Scale AI 2026).

ARC-AGI-2 moved quickly too. Its May 2025 technical report placed leading model baselines at less than 5 percent (Chollet et al. 2025). DeepMind reported an ARC Prize-verified score of 84.6 percent in February 2026 (Google DeepMind 2026). The model result and the reported human average came from different evaluation sets that were calibrated for similar difficulty, not from identical items. Compute, cost, and access conditions also differ. The 2026 competition still reserves a bonus threshold at 85 percent on a fully private set. Under that protocol, the benchmark is not solved.

Construct validity asks an earlier question: what ability does the score claim to measure? A NeurIPS review had 29 experts examine 445 benchmark papers and ended with eight recommendations. It found recurring gaps in construct definition, real-world task coverage, statistical testing, and validity arguments; it did not show that every benchmark measures nothing (Bean and others 2025). A complementary General Scales study demonstrated 18 demand rubrics across 15 language models and 63 tasks, using instance annotations to explain and predict difficulty. That is a promising tested method, not yet a universal replacement for evaluation (Zhou et al. 2026).

Exploitability is more direct. BenchJack's 2026 preprint audited ten agent benchmarks, catalogued 219 flaws, and synthesized near-perfect exploits on nine of ten without solving the intended tasks (Wang et al. 2026). This was a deliberately adversarial audit with privileged knowledge of the benchmarks. It does not imply that ordinary model runs always cheat. It does show that a score needs an adversarial audit before it can support a broad capability claim.

The instruments saturate faster than they can be built in some fast-moving domains, but saturation is not the only failure and is not inevitable for every test. A replacement benchmark helps only if its construct, protocol, and resistance to shortcuts improve with its difficulty.

Forecasts are scenarios, not measurements

Trend extrapolation is useful when its assumptions stay visible. One exploratory eight-parameter model projected more than 99 percent automation of AI research and development tasks around 2032. Its author said the parameter values were educated guesses, the model and write-up took roughly 15 hours, there was no backtest, and he put little weight on the exact date (Kwa 2026). This is a scenario, not a measurement and not an institutional consensus.

The durable progress versus compute-bought artifact debate cannot be settled by choosing a favorite fit. A continuation case needs evidence that gains persist across frozen tasks, system budgets, and reliability thresholds. A constraint case needs evidence that the trend bends when a named resource binds. Changes in reinforcement learning, scaffolds, and test-time compute are candidate explanations, not conclusions inferred from the horizon alone. The resource mechanics belong to Chapter 5.

Keep a measurement ledger

A frontier claim is useful only when its contract travels with it. For every result, record at least:

This ledger does not make one benchmark definitive. It makes comparisons auditable. It also tells an organization when a result has expired because the system, tasks, or deployment changed.

The frontier is therefore a set of moving, conditional limits. Horizons reveal progress that saturated quizzes can miss; economic tasks bring evaluation closer to work; controlled studies test whether the value survives deployment. None is interchangeable with the others. Chapter 72 takes the next step: when a model can produce work that an ordinary reviewer cannot cheaply verify, capability measurement gives way to acceptance and control.

Further reading