Inference-Time Scaling

Once a model's weights are fixed, a system can still spend more work on an individual prompt. It can generate additional candidates, extend or revise a partial answer, call a tool, search a branching state space, or evaluate several alternatives before returning one. These operations are grouped under inference-time scaling, but they do not spend the same resource or improve an answer for the same reason.

The model is only one component of this process. A controller decides what work to request, an evaluator supplies evidence about the results, and a stopping rule decides when further work is no longer worth its cost. More computation helps only when that system can create useful alternatives and identify or construct a better answer from them.

What is being scaled

An inference strategy can vary several quantities at once:

• Breadth: the number and diversity of candidate responses.
• Depth: the length of a trajectory or the number of dependent revision steps.
• Evaluation: tests, proof checks, learned scores, answer aggregation, or judge calls applied to candidates and partial states.
• External work: retrieval, code execution, simulation, or other tools.
• Serving resources: model choice, accelerator time, memory, concurrency, and the latency allowed for the request.

Parallel and sequential work describe dependencies, not two exclusive method families. Independent samples can run concurrently. A revision must wait for the answer it revises. Beam search, lookahead, and tool-using agents mix both: they branch at some steps, evaluate the branches, and continue from selected states. A complete description therefore names the proposal method, evaluation rule, controller, and budget instead of reporting only “reasoning tokens.”

flowchart TD
    A[Prompt and task policy] --> B[Inference controller]
    B --> C[Generate candidates or revisions]
    B --> D[Call tools or retrieval]
    C --> E[Candidate set or partial state]
    D --> E
    E --> F[Evaluate and aggregate evidence]
    F --> G{Stop rule}
    G -->|accept| H[Return answer]
    G -->|continue| B
    G -->|escalate| I[Stronger model or human path]
    J[Cost, latency, and concurrency limits] --> B

A production controller is solving a constrained decision problem. One useful form is

$$a^*(x) = \arg\max_{a\in\mathcal A} \left( \mathbb E[U_x(Y_a)] -\lambda\,\mathbb E[C_a] -\mu\,\mathbb E[L_a] \right).$$

Here $x$ is the prompt; $\mathcal A$ is the available set of inference strategies; $Y_a$ is the answer returned by strategy $a$; and $U_x$ is the true task utility, including any value assigned to abstaining or escalating. $C_a$ denotes compute or monetary cost, $L_a$ latency, and the nonnegative weights $\lambda$ and $\mu$ express the deployment's cost and latency trade-offs. The router cannot observe true utility before answering. It must estimate the value of each action from held-out data and signals available at runtime.

This objective also shows why inference compute is not automatically cheaper than training or model capacity. Some strategies require a separately trained verifier or revision model. The best choice depends on request volume, hardware, quality targets, and which prompts can benefit from additional work.

Repeated sampling buys coverage

Parallel sampling is the simplest case. Draw a response $Y$ from the fixed proposal distribution $\pi(\cdot\mid x)$ for prompt $x$. Here $h_x(Y)\in\{0,1\}$ indicates whether that response satisfies the task's true acceptance condition. The one-sample success probability is

$$p_x = \Pr_{Y\sim\pi(\cdot\mid x)}[h_x(Y)=1].$$

Now draw $k$ responses that are conditionally independent and share the same success probability. Their coverage means the probability that at least one of those responses is correct:

$$C_k(x)=1-(1-p_x)^k.$$

In these formulas, $p_x$ measures one-sample success for this prompt and proposal, while $C_k(x)$ measures whether a correct response exists among the $k$ samples. Coverage increases monotonically when $p_x>0$, but with diminishing returns. The formula is exact for independent draws from an unchanged proposal. Independence can fail when later prompts adapt to earlier outputs, calls share state, or a controller couples, deduplicates, or resamples candidates. Diversity is a separate issue: independent draws can still repeat the same answer because repetition is part of $\pi(\cdot\mid x)$. Sampling temperature, prompt variation, model diversity, and search policy all change the proposal and therefore its coverage.

Brown et al. measured coverage across models and tasks for as many as 10,000 samples per problem. Aggregate curves were often well fit by an exponentiated power law, though the fit was not universal. In their SWE-bench Lite experiment, DeepSeek-Coder-V2-Instruct solved 15.9% of issues with one sample and 56% with 250 samples when candidate patches could be tested (Brown et al. 2024). That is evidence about a particular proposal, benchmark, and verification setup. It does not imply that every task has a comparable scaling curve.

Selection determines realized accuracy

Coverage is not the accuracy of a deployed system. Suppose a selector $S$ must return one of the sampled candidates $Y_{1:k}$. Its realized accuracy is

$$A_k(x) = \Pr\!\left[ h_x\!\left(Y_{S(Y_{1:k})}\right)=1 \right].$$

Here $S(Y_{1:k})$ is the index chosen from the $k$ candidates, and $A_k(x)$ is the probability that the selected candidate passes the true task condition. For a selector restricted to the sampled set, $A_k(x)\le C_k(x)$. Equality requires an exact procedure that returns a correct candidate whenever one is present. A controller that revises or synthesizes a new answer is not subject to this particular bound because its output can leave the original set.

Selectors have different evidence and different failure modes:

• An exact checker validates a fully specified property. A proof kernel can establish that a proof term satisfies its formal rules. A test suite usually checks only sampled behavior, so a passing program can still be wrong outside the tests. Verification also consumes compute.
• Answer voting groups candidates by a normalized final answer and chooses the largest group. Self-consistency introduced this sample-and-marginalize procedure for reasoning paths and evaluated up to 40 paths (Wang et al. 2023). Voting works when probability mass concentrates on the correct answer, not merely because one correct sample exists.
• A learned scorer or judge ranks candidates using a proxy for task utility. It can distinguish answers that do not admit exact checks, but its errors may be correlated with the generator and can be exploited by a large candidate pool.

In Brown et al.'s GSM8K and MATH selection experiments, majority voting and learned reward-model selection failed to track the continued growth in oracle coverage (Brown et al. 2024). More generally, best-of-$n$ selection against an imperfect reward model can eventually prefer responses that exploit the model's scoring errors. Huang et al. formalize this failure and show why an ideal candidate count can exist when the reward model is imperfect (Huang et al. 2025). A larger pool is therefore useful evidence only if the selection rule remains reliable over the distribution induced by that larger pool.

The following runnable is a constructed failure model, not an empirical fit. Each candidate is correct with probability p. Each wrong candidate has probability selector_bias of receiving a misleadingly high mean score, and Gaussian noise has standard deviation noise. Increasing k always raises coverage, but it also creates more opportunities for a misleading wrong candidate to win.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)
p = 0.30
selector_bias = 0.05
noise = 0.40
trials = 4000
ks = range(1, 65)
coverage, selected_accuracy = [], []

for k in ks:
    correct = rng.random((trials, k)) < p
    misleading = (~correct) & (rng.random((trials, k)) < selector_bias)
    mean_score = np.where(misleading, 1.5, np.where(correct, 1.0, 0.0))
    score = mean_score + rng.normal(0, noise, (trials, k))
    coverage.append(np.mean(correct.any(axis=1)))
    chosen = score.argmax(axis=1)
    selected_accuracy.append(np.mean(correct[np.arange(trials), chosen]))

plt.plot(list(ks), coverage, label="coverage")
plt.plot(list(ks), selected_accuracy, label="imperfect selector")
plt.xlabel("candidates k")
plt.ylabel("fraction correct")
plt.legend()
plt.show()
print("best selector k:", list(ks)[int(np.argmax(selected_accuracy))])

Sequential work needs a feedback source

Sequential scaling makes later work depend on earlier state. A model may extend one trace, critique a draft, revise after a failed test, or expand a search node chosen from a frontier. A generic revision loop can be written as

$$Y_0\sim\pi(\cdot\mid x),\qquad F_t=\phi(x,Y_t,e_t),\qquad Y_{t+1}\sim R(\cdot\mid x,Y_t,F_t).$$

Here $Y_t$ is the answer at revision step $t$; $e_t$ is the evidence available at that step; $\phi$ converts the prompt, answer, and evidence into feedback $F_t$; and $R$ is the revision distribution. The evidence may be the model's own critique, an executable check, a tool result, retrieved material, or human feedback. These sources should not be described as equivalent. They differ in cost, independence, and how directly they constrain correctness.

The s1 work provides a narrow example of direct length control. Its budget forcing procedure could suppress the end-of-thinking token and append “Wait” when s1-32B tried to stop, or terminate thinking at a fixed limit. On AIME 2024, the final paper reports an increase from 50% without the intervention to 57% with it (Muennighoff et al. 2025). AIME 2024 contains only 30 questions, so each question moves the score by about 3.3 percentage points. This result combines a particular model, its 1,000-example SFT set, a competition-math benchmark, and a decoding procedure. It does not establish that appending a continuation cue improves arbitrary models or that accuracy rises monotonically with trace length.

Revision without new evidence is especially fragile. Huang et al. define intrinsic self-correction as asking a model to correct its answer without external feedback. Across their reasoning experiments, the studied models struggled to improve and sometimes degraded their answers (Huang et al. 2024). A unit-test failure or a retrieved fact changes the information available to the next step; a second opinion from the same model may only restate the first error. Sequential compute is valuable when the loop supplies a useful search operation or a feedback channel, not simply because the transcript is longer.

Allocate a budget instead of maximizing it

A uniform budget wastes work on easy prompts and may still be insufficient for hard ones. Snell et al. studied two mechanisms on MATH using PaLM 2 models fine-tuned for the relevant operation: search against a process reward model and sequential revision of the proposal distribution. The effective strategy depended on both problem difficulty and the base model (Snell et al. 2025). Their difficulty bins used 2,048 samples per problem and either ground-truth pass@1 or the verifier's average score, while the cost of estimating difficulty was excluded. Their compute-optimal allocation was more than four times as efficient as a fixed best-of-$n$ baseline in the reported setup. In parts of the paper's FLOPs-matched workload analysis, the smaller model with test-time compute outperformed a model about fourteen times larger only on problem subsets where the smaller model already had a nontrivial success rate; the comparison also depended on the assumed ratio between inference and pretraining demand.

Those findings are conditional, not a theorem that inference compute replaces parameters. The study estimated problem difficulty using large offline sample pools and a learned verifier; that oracle-like information is not free at deployment. A practical router must infer difficulty from cheaper signals such as a first-pass score, candidate disagreement, checker failures, prompt class, or historical outcomes. It must also recognize uncertainty in that estimate. No observed success within one proposal and budget does not prove that the true success probability is zero. A different model, prompt, tool, or decomposition may create coverage that repeated sampling from the original proposal did not.

Longer computation can also reverse an initially correct answer. A 2026 study of R1-32B and s1-32B across AIME 2024/2025, MATH-500, and GPQA Diamond used forced budgets from 500 to 16,000 tokens. It found diminishing marginal returns, cases where models abandoned previously correct answers, and different useful stopping lengths for different difficulty levels (Zhou et al. 2026). The appropriate response is not one universal token cap. It is a validated stopping policy with both a hard resource limit and task-specific evidence for continuing.

Account for the work where it is paid

Generated tokens are convenient to count, but they are not a complete compute measure. A useful request-level ledger over $R$ attempts is

$$C_{\mathrm{total}} = \sum_{r=1}^{R} \left( C_{\mathrm{gen},r}+C_{\mathrm{eval},r}+C_{\mathrm{tools},r} \right) +C_{\mathrm{controller}},$$

where attempt $r$ may be an initial call or retry; its three terms denote candidate generation, evaluation, and external tool work; and $C_{\mathrm{controller}}$ is routing and orchestration overhead. End-to-end latency follows the workflow's critical path rather than the sum of all work:

$$L_{\mathrm{e2e}} = L_{\mathrm{queue}}+L_{\mathrm{prefill}} +\max_{p\in\mathcal P}\sum_{s\in p}L_s.$$

Here $\mathcal P$ is the set of dependency paths through generation, evaluation, and tool stages; $s$ is one stage on path $p$; and $L_s$ is that stage's latency. Queueing and prompt prefill precede the longest dependent path. Parallel samples can overlap in wall-clock time only when spare concurrency exists. Under a fixed hardware budget they may increase queueing, reduce batching efficiency, or delay other users even while one request finishes sooner.

Comparisons should therefore match the resource that matters. Equal output tokens do not imply equal floating-point operations when model sizes differ. Equal floating-point operations do not imply equal latency when one strategy is serial and another uses wide concurrency. Equal latency does not imply equal fleet cost. Report generated and evaluated tokens, model and verifier calls, tool executions, peak concurrency, accelerator time, and latency percentiles alongside task quality.

Operating an inference policy

A defensible deployment makes the controller observable and testable:

1. Define utility before routing. Specify task success, acceptable abstention, escalation outcomes, cost, and latency limits for each request class.
2. Measure complete curves. On held-out prompt families, sweep candidate count, revision depth, model choice, and evaluation effort. Record quality, total work, tail latency, and variance rather than one preferred setting.
3. Compare matched strategies. Give parallel sampling, sequential revision, stronger-model escalation, and tool use the same cost or latency envelope.
4. Audit selectors under search. Measure false accepts, false rejects, calibration, and performance as the candidate pool grows. Include adversarial candidates selected to expose scorer weaknesses.
5. Use explicit stop reasons. Log exact-check success, stable agreement, budget exhaustion, low expected marginal value, timeout, and escalation as distinct outcomes.
6. Meter every component. Store generation, evaluation, tool, retry, and queueing costs per request, together with the controller version and model checkpoints.
7. Protect the budget boundary. Apply hard caps and rate limits even when an adaptive model requests more work. Treat retrieved or user-supplied content as untrusted input to the controller.

The last control is a security requirement as well as a cost control. The OverThink study inserted decoy reasoning problems into content used by retrieval-augmented systems. Across its evaluated models, it reported slowdowns up to 18 times on FreshQA and 46 times on SQuAD while preserving contextually correct answers (Kumar et al. 2025). Output-only monitoring would miss that failure. Reasoning work, tool calls, and wall-clock occupancy need their own alerts and quotas.

Payoff and boundary

Inference-time compute can turn a frozen model into a stronger system for some prompts. Its value comes from a complete policy: create alternatives, gather evidence, choose or revise an answer, and stop before the marginal benefit falls below the resource cost. Extra candidates without selection buy coverage, not a returned answer. Extra revisions without a validated improvement signal add work; they do not establish reliable correction. A scaling claim is operationally meaningful only when it reports quality together with generation, evaluation, tools, latency, and the limits of the measured regime.

Further reading