The Inference-Time Scaling Paradigm

Training in Chapter 28 spends compute once to bake reasoning into the weights. Inference-time scaling is the other lever: after the weights are frozen, spend more compute per prompt to make a fixed model answer harder questions. Repeated sampling buys coverage but not an answer; sequential revision and parallel search are different shapes of test-time compute with different ideal regimes; compute-optimal allocation routes a budget by prompt difficulty; and the whole curve bends back down when the selector is imperfect.

The one knob left after training

A trained model has one knob left at inference: how much compute to spend per prompt. The cheapest answer is one greedy decode. The question is what spending more buys: a thousand samples instead of one, or a long deliberate chain instead of a short one, and whether that spend is ever a better deal than training a larger model in the first place.

The reason the question is live is that the two spends are fungible. A larger model costs more to train and more per token forever. Test-time compute costs nothing at training and is paid only on the prompts that need it, which is most of the value on a long tail of hard prompts and none of it on the easy majority. If a small model plus extra inference can match a large model on the hard prompts, the serving economics of Chapter 31 change, because you move cost from a fixed asset you pay for on every request to a variable you spend only when a prompt earns it.

Two shapes of test-time compute

Test-time compute comes in two shapes, and the distinction organizes the whole chapter. Parallel scaling draws many independent samples and combines them. Sequential scaling spends the compute in one growing chain, where each step conditions on the last, revising or extending a single line of thought. Snell et al. frame these as revising the proposal distribution sequentially versus searching against a verifier in parallel, and the central empirical claim is that neither dominates: the better shape depends on the prompt (Snell et al. 2024).

Sequential scaling spends the budget the second way. Instead of $k$ independent shots, the model keeps thinking in one trajectory, and the compute buys length and revision rather than breadth. The reasoning models of Chapter 28 already do this natively: a long chain of thought is sequential test-time compute that the RL of that chapter taught the model to produce. s1 shows the lever in its barest form with budget forcing, a decoding-time intervention that controls how long the model thinks by suppressing the end-of-thinking token and appending the word "Wait" to push the model to continue, or by forcing termination to cap the spend (Muennighoff et al. 2025). Lengthening the chain this way lets the model catch and fix its own errors, and accuracy rises with the forced budget. Sequential and parallel are composable: you can draw several long chains and select among them, and Snell et al. study exactly this two-dimensional budget (Snell et al. 2024). Figure 30.1 lays out the two shapes and where the selector decides whether coverage becomes accuracy.

Coverage is not accuracy

The parallel story starts with the simplest possible method. Draw $k$ samples and ask what fraction of problems are solved by at least one of them. Brown et al. call this coverage, and it is the ceiling on what any selection rule could extract from the samples (Brown et al. 2024). Coverage rises smoothly with $k$ across four orders of magnitude, well modeled as an exponentiated power law (a smooth curve that climbs toward a ceiling with predictable diminishing returns as $k$ grows), which is the inference-time analogue of the training scaling laws in Chapter 5 (Brown et al. 2024). The key word is coverage, not accuracy. A correct sample exists in the set; that is not the same as knowing which one it is.

That gap is the design problem of parallel scaling. To turn coverage into an answer you need a selector, and the selector is where the difficulty lives. When the task has an automatic verifier, the unit tests and answer checkers and proof checkers of Chapter 28, selection is free: run the verifier on each sample and keep one that passes. Coverage converts directly into accuracy, which is why repeated sampling pays most in code and formal math. When there is no automatic verifier, you fall back to a heuristic selector, majority vote over the answers or a learned reward model (a model trained to score how good an answer looks) scoring each sample. These work to a point and then plateau: Brown et al. report that majority vote and reward models stop scaling past a few hundred samples even as coverage keeps climbing, because the selector cannot tell the rare right answer from the many confident wrong ones (Brown et al. 2024). The design problem turns on this one gap between a correct answer existing and a system finding it. Figure 30.2 makes the gap visible: coverage climbs smoothly with $k$, a verifier realizes almost all of it, and a heuristic selector tracks coverage early and then plateaus far below it.

Spending compute on the check

If the selector is the bottleneck, the question becomes how much compute to spend checking rather than generating. The reason any of this pays is an asymmetry: for many tasks, deciding whether a candidate is right is cheaper and more reliable than producing it, so a verifier need not match the generator's strength to be useful. Generative verifiers push the idea further. Rather than a frozen scalar head, generative reward model (GenRM) trains the verifier with the same next-token objective as the generator, jointly on verification and generation, so the verifier can write a chain of thought about a candidate and even spend its own test-time compute by voting over several such judgments (Zhang et al. 2024). The selector becomes a second reasoner, not a fixed score.

That asymmetry opens two deployments that pull in opposite directions. A cheap verifier can gate an expensive generator, rejecting obviously wrong candidates before paying for more samples or a longer chain. And a strong verifier can lift a weaker generator above what it could select for itself, the inference-time face of the weak-to-strong question in Chapter 55. The cheapest verifier of all is the model itself. Self-Refine loops generate, critique, and revise with a single model and no extra training (Madaan et al. 2023). In Reflexion, a failure signal becomes a natural-language reflection held in an episodic memory and fed into a retry; the method reports large gains on coding tasks (Shinn et al. 2023). Self-RAG goes further inside the model, training it to emit reflection tokens that decide when to retrieve and whether its own output is supported (Asai et al. 2023).

The catch is that a self-check is only as good as the signal behind it. Huang et al. find that intrinsic self-correction, revising on the model's own judgment with no external feedback, fails on reasoning and can lower accuracy, because a model confident enough to assert a wrong answer is usually confident enough to wave it through on a second look (Huang et al. 2024). The self-correction methods that work route a real signal into the loop: a unit test, a retrieval, a tool result, a verifier that checks against ground truth. This is the inference-time echo of Chapter 28, where a checkable reward beat a learned proxy, and it is what the agent loop of Chapter 41 spends itself on.

From Methods to a Compute Axis

The methods are older than the paradigm. self-consistency, sampling several chains and taking a majority vote over final answers, is parallel scaling with a majority selector and predates the reasoning-model era (Wang et al. 2022). Best-of-$n$ against a reward model is parallel scaling with a learned selector. A process reward model (PRM) (process reward model), trained in "Let's Verify Step by Step" and used as the dense verifier in Chapter 28, is what makes verifier-guided search over partial chains work, scoring steps rather than only final answers so a search can prune early (Lightman et al. 2023). What changed in 2024 was the framing: these stopped being isolated methods and became a compute axis with its own scaling laws.

Brown et al. established the parallel law: coverage as a smooth, predictable function of sample count, and the sharp dependence of its payoff on whether a verifier exists (Brown et al. 2024). Snell et al. asked the allocation question. Given a fixed test-time budget, how should you split it, more parallel samples, more sequential revision, a deeper verifier search (Snell et al. 2024).

Routing the budget by difficulty

Snell et al. showed the answer depends on prompt difficulty. Easy prompts benefit from sequential refinement of an already-good first guess; hard prompts benefit from broader parallel search. Routing the budget by difficulty, their compute-optimal strategy, beat a best-of-$n$ baseline by more than four times in efficiency, and on prompts where a small model already had a foothold, a FLOPs-matched small model with extra test-time compute outperformed a model roughly fourteen times larger (Snell et al. 2024). That is the constraint made quantitative: inference compute substituting for parameters. Figure 30.3 traces how the budget is routed by difficulty.

What Snell et al. framed as advice became serving architecture in August 2025: GPT-5 ships as a fast model, a thinking model, and a per-request router that picks between them, with reasoning-effort levels exposed in the API, and Anthropic and Google sell thinking budgets the caller sets per request (OpenAI 2025).

Less data, more inference

The s1 and LIMO results sharpened a different edge. s1 reached strong competition-math performance by supervised fine-tuning on only about a thousand carefully chosen reasoning traces and then scaling at test time with budget forcing (Muennighoff et al. 2025). LIMO made the less-is-more claim explicit with a few hundred examples, arguing that when a base model already encodes the knowledge from pre-training, a small set of high-quality demonstrations is enough to elicit the reasoning, and the heavy lifting moves to inference (Ye et al. 2025). Both point the same way: the capacity may already be latent in the weights, and test-time compute is how you call it out. This connects to the contested question in Chapter 28 about whether RL teaches new reasoning or elicits what is already there, seen now from the inference side.

import numpy as np
import matplotlib.pyplot as plt
rng = np.random.default_rng(0)
p, selector_bias, noise, trials, ks = 0.3, 0.05, 0.4, 4000, range(1, 65)
cov, heuristic = [], []
for k in ks:
    correct = rng.random((trials, k)) < p                 # which samples are right
    confident_wrong = (~correct) & (rng.random((trials, k)) < selector_bias)
    base = np.where(confident_wrong, 1.5, np.where(correct, 1.0, 0.0))
    score = base + rng.normal(0, noise, (trials, k))      # selector score, noisy
    cov.append(np.mean(correct.any(axis=1)))              # a right answer exists
    picked = score.argmax(axis=1)                          # selector keeps top score
    heuristic.append(np.mean(correct[np.arange(trials), picked]))
plt.plot(list(ks), cov, label="coverage / perfect verifier")
plt.plot(list(ks), heuristic, label=f"imperfect selector (bias={selector_bias})")
plt.xlabel("samples k"); plt.ylabel("accuracy"); plt.legend(); plt.show()
print("best heuristic k:", list(ks)[int(np.argmax(heuristic))])

The balances and their knees

The paradigm is a set of balances, each with a knee.

• Parallel versus sequential. Parallel sampling is embarrassingly concurrent and latency-friendly, you can decode $k$ samples at once, but it needs a selector and wastes compute on redundant near-duplicates. Sequential revision uses compute more efficiently on prompts where a first guess is close, but it is inherently serial and pays full latency. Snell et al. show the right mix is difficulty-dependent, not a constant (Snell et al. 2024).
• Coverage versus selection. More samples always raise coverage, but the realized accuracy is capped by the selector. With a verifier the cap is the coverage ceiling; without one the cap is wherever majority vote or the reward model plateaus. Spending past that point buys nothing.
• Test-time compute versus model size. Extra inference can substitute for parameters on the hard tail, shifting cost from a fixed training-and-serving asset to a per-prompt variable. But the substitution has a regime: it works when the small model already has nonzero coverage, and it fails on prompts the small model never solves no matter how many tries, where only a stronger model or better training helps.
• Data efficiency versus compute. s1 and LIMO trade training data for test-time compute, eliciting latent capability with a tiny SFT set plus inference scaling (Muennighoff et al. 2025; Ye et al. 2025). The bet holds only when the base model already encodes the domain; on genuinely new capability there is nothing latent to elicit.
• Latency and cost versus accuracy. Every method here multiplies the per-prompt compute and, for sequential methods, the latency. The product decision is which prompts deserve the spend, which is exactly what compute-optimal allocation formalizes.

Overthinking as a Production Failure

The reasoning budget has a second knee: after a point, more thinking can hurt quality or simply burn the serving budget. Ghosal et al. report a non-monotonic pattern in reasoning models, where additional thinking first improves accuracy and then degrades it, motivating parallel thinking and adaptive stopping rather than unbounded chain extension (Ghosal et al. 2025). This matches the product experience of reasoning models: a hard prompt may deserve a long trace; a simple prompt that triggers the same trace is waste.

That waste is now an attack surface. OverThink shows that decoy reasoning problems injected into retrieved content can make reasoning models spend many more reasoning tokens while still returning contextually correct answers, with large slowdowns reported across closed and open reasoning models (Kumar et al. 2025). Excessive Reasoning Attack crafts inputs that delay termination and trigger extra reasoning paths, increasing reasoning length by several times while preserving task utility (Si et al. 2025). These attacks do not need to make the final answer wrong. They make the system too expensive or too slow.

The operational response is to treat reasoning tokens as metered, routable work. Give each task class a default effort, the knob that providers now expose directly as reasoning-effort levels and thinking budgets, and estimate difficulty before committing the full budget. Stop early once the verifier is confident, and cap sequential thinking even when the model asks to continue. Suspicious retrieved content should run through a cheap filter before it reaches a slow reasoner. And log the hidden cost separately from output tokens, so overthinking appears in observability rather than only in the cloud invoice. The concise-and-adaptive thinking literature collects this as its own design problem: reasoning models need to learn when to think less, not only how to think longer (Zhu and Li 2025).

Building it

The operational core is small. A parallel scaler is a sampling loop plus a selector: draw $k$ completions at a temperature high enough for diversity, then either run the verifier and keep a passing one, or take a majority vote, or score with a reward model and take the best. The verifier path is the one that scales, so the engineering effort goes where Chapter 28 put it, into making the checker sound and broad. A sequential scaler is a controlled decode: detect when the model tries to stop and decide whether to let it, the budget-forcing intervention, suppressing the stop token and injecting a continuation cue to extend, or forcing a final answer to cap.

# Parallel scaling: coverage is free, selection is the hard part.
def best_of_k(prompt, k, select):
    samples = [model.sample(prompt, temperature=0.8) for _ in range(k)]
    return select(samples)  # verifier > reward model > majority vote

The failure modes follow from the design. The central one is selecting with an imperfect verifier: more samples feed a flawed selector more chances to pick a confident wrong answer, the inference-time reward hacking named in the contested box, so the curve flattens or dips and the extra compute is wasted. A related trap is coverage without a selector, where the right answer is provably in the set and the system still returns a wrong one, which looks like a model failure but is a selection failure. Then there is sequential over-thinking: budget forcing past the useful point can make a model second-guess a correct answer into a wrong one, so the forced budget itself has a knee. And the substitution fails silently on prompts of zero coverage, where no amount of test-time compute helps because no sample is ever correct, and the only fix is upstream in the model.

The selector is where these threads meet. Test-time scaling is, first, a capability lever: it turns a frozen model into a stronger one on the hard tail without retraining. As an efficiency lever, it moves cost off the fixed asset and onto the prompts that earn it, at the serving prices set above. And it is a trust hazard, because the entire payoff rides on the selector, and one you do not trust converts more compute into more confidently wrong answers. The verifier quality that bounded the training loop in Chapter 28 bounds the inference loop here too: the same ground truth that makes RL safe is what makes test-time scaling pay.

Further reading