Eliciting Reasoning

A base model already holds far more reasoning ability than greedy decoding reveals, and you can draw it out at inference time without touching a single weight. Elicitation is the family of prompting and decoding methods that turns one forward pass into a deliberation. A chain of thought (CoT), a written chain of intermediate reasoning steps, helps because it gives intermediate work tokens to live in. Sampling many chains and voting works for a different reason: wrong paths scatter. Trees and searches pay only when the problem has useful branches, while a verifier is the component that turns best-of-n from raw sampling into selection. Teaching the weights to reason is the sibling problem of Chapter 28; here nothing is trained.

The answer token comes too early

Ask a model a multi-step question and decode greedily, and it commits to an answer token before it has done the work that answer depends on. The autoregressive objective of Chapter 5 spends a fixed, equal slice of compute on every token, so a problem that needs ten steps of arithmetic gets the same single pass as a problem that needs none. The model has the sub-skills, since pre-training saw the additions, the lookups, and the deductions, but the decode gives them nowhere to run. The answer position is asked to integrate everything at once.

That observation sets the design problem. The work is to spend more inference compute, and spend it in the right shape, so that latent ability surfaces, without changing the model. Two levers exist at inference time. One is the prompt, which can ask the model to externalize intermediate steps into tokens it can then condition on. The other is the decoding procedure, which can sample, branch, score, and select instead of greedily taking the top token. The methods that follow combine those two levers, and they arrived in a clear lineage: first give the reasoning somewhere to run, then stop trusting one sample of it, then shape and search the space it opens. Figure 24.2 previews the four shapes that lineage produces.

Give the reasoning somewhere to run

The first move is to make the model write its working, then exploit that the working is now part of the context. A chain-of-thought is the minimal form: prompt the model to produce intermediate reasoning steps before the final answer, and each step becomes a token sequence the later steps attend over. This converts a single hard prediction into a sequence of easier ones, and it gives the model room to allocate more forward passes, one per reasoning token, to a harder problem. A few worked examples in the prompt elicit this (Wei et al. 2022), and even the bare instruction "Let's think step by step" elicits it zero-shot, with no exemplars at all (Kojima et al. 2022). No weights move. The ability was latent; the prompt unlocks the decode, and the zero-shot result makes the point sharp: the effect is about eliciting, not demonstrating.

Stop trusting a single sample

Once reasoning lives in tokens, the decoding procedure becomes a design space. The next move is to stop trusting a single sample. A greedy chain can take a wrong turn early and never recover. self-consistency samples many chains at non-zero temperature, then takes a majority vote on the final answers (Wang et al. 2022). The reasoning path is treated as scratch work to sum over, not an output to return:

$$\hat{y} = \arg\max_{y} \sum_{i=1}^{n} \mathbf{1}\left[ a(z_i) = y \right], \quad z_i \sim p_\theta(z \mid x)$$

where each $z_i$ is a sampled chain and $a(z_i)$ is the answer it reaches. Many paths can reach the right answer by different routes, while wrong answers tend to scatter, so the mode of the answer distribution is a better estimate than any one sample. This is the highest value-per-line method in this family: it needs no verifier, no extra model, only more samples and a vote, and it remains the strongest simple baseline for exactly that reason (Wang et al. 2022).

Try this: each chain is independently correct with probability p, and the simulation votes over n chains. Watch majority-vote accuracy rise above a single sample once p > 0.5, then saturate.

import numpy as np

rng = np.random.default_rng(0)
p = 0.55           # per-chain probability of the correct answer
trials = 20000     # Monte Carlo trials per n
ns = [1, 3, 5, 11, 21, 41]

for n in ns:
    correct = rng.random((trials, n)) < p   # which chains got it right
    votes = correct.sum(axis=1)             # how many voted for the right answer
    win = (votes > n / 2).mean()            # majority is right (ties broken as wrong)
    print(f"n={n:>2}  majority-vote accuracy = {win:.3f}")

print(f"\nSingle chain (n=1) accuracy = {p:.3f}; voting amplifies a weak edge.")

Shape the chain, then search the space

Sampling more of the same chain is not the only refinement. When a problem decomposes, you can shape the chain instead of just sampling it. least-to-most prompting asks the model first to break a problem into ordered subproblems and then to solve them in sequence, each conditioned on the solved ones, which lets the model generalize to problems harder than any single exemplar in the prompt (Zhou et al. 2022).

The richest design treats reasoning as search over a space of partial solutions. tree of thoughts (ToT) makes each node a coherent intermediate thought, expands several children per node, has the model self-evaluate each as promising or not, and explores the tree with breadth-first or depth-first search, including lookahead and backtracking (Yao et al. 2023). A chain is then the degenerate tree with one child per node and no backtracking. Pushing this further, AlphaZero-style methods run Monte Carlo Tree Search over the reasoning tree, using value estimates to decide where to expand, so compute concentrates on promising branches rather than spreading evenly (Feng et al. 2023).

The chain, the tree, and the value-guided search differ in the shape of the space they explore, shown in Figure 24.3. A chain walks one line. A tree fans out, self-evaluates each node, and can backtrack out of a dead end. Value-guided search goes one step further, adding a value estimate so compute concentrates on promising branches instead of spreading evenly.

When a vote is not enough: the verifier

Self-consistency votes, but a vote only works when the answer is a discrete label to count. Best-of-n replaces the vote with a score. Sample $n$ candidate solutions, score each with a verifier or reward model (a ground-truth checker, or a learned scorer of quality), and return the highest. This was introduced for grade-school math: train a verifier to judge candidate solutions, generate many, and keep the one the verifier ranks highest, which beat a fine-tuning baseline (Cobbe et al. 2021). The same scorer can guide the search rather than just rank the finished output, evaluating partial states inside a tree, which is what self-evaluation guided beam search does (Xie et al. 2023). The verifier is the component that turns raw samples into a selected answer, and its quality is the ceiling on what selection can buy. That ceiling, and how it bounds test-time scaling, is the subject of Chapter 30.

A vote and a score can disagree, and Figure 24.5 shows when. Over the same sampled chains, a vote rewards agreement and a score rewards correctness. When a biased model is consistently wrong, the majority answer is also wrong, and only a verifier that checks the answer against ground truth can rescue the minority chain that happens to be right.

What each shape costs, and what bounds it

Each method buys accuracy with inference compute, and the exchange rate differs.

• Single chain versus voting. One chain is cheapest and lowest latency but brittle to an early wrong turn. Self-consistency multiplies token cost by the sample count $n$ for a robust majority answer, with diminishing returns as $n$ grows. It needs a discrete, comparable answer, so it does not apply to open-ended generation where there is nothing to vote on.
• Voting versus scoring. A vote needs no extra model but cannot rank free-form outputs and cannot tell a confidently-wrong consensus from a correct one. Best-of-n needs a verifier or reward model but ranks anything that scorer covers, and its ceiling rises with verifier quality. A vote rewards agreement; a score rewards correctness, and they diverge when the model is consistently wrong.
• Chain versus tree versus search. A chain spends compute in one line. A tree spends it on breadth and can backtrack out of dead ends, at the cost of many self-evaluation calls per node, which are themselves model calls that can be wrong. MCTS-style search concentrates compute on high-value branches but adds the machinery and the dependence on a value estimate that, if miscalibrated, steers the search wrong. The gain over self-consistency is real but smaller than its cost on many tasks, so the simpler method is often the right default.
• Compute now versus training once. Every method here pays its cost on every query, forever. Training the behavior in, per Chapter 28, pays once and amortizes, but costs a training run and the data to do it. Elicitation is the right tool when the model is fixed, the task is new, or the volume is low; internalization wins at scale.

Two structural boundaries keep elicitation in check, and neither is a tuning knob. It cannot exceed what the fixed model can produce, only surface and select among it, so a problem the model never solves in any sample is out of reach no matter how many you draw or how hard you search. And the visible chain is a lever on accuracy, not a guaranteed account of the computation, so reading it as oversight, per Chapter 55, is a claim that needs its own test, not a free byproduct of asking the model to think out loud.

Stacking them, and the move back to training

The methods compose more than they compete, and the practical recipe stacks them. Chain-of-thought is the substrate: get the model to externalize steps, by exemplars or by a step-by-step instruction. On top of it, self-consistency is the first upgrade, sample a handful to a few dozen chains and vote, because it is cheap to add and needs no other component. When a verifier exists, best-of-n over the same samples replaces the vote with a score and lifts the ceiling. A tree or a search is the heaviest tool, justified when the problem has a real branching structure and dead ends that backtracking escapes, and when the per-node self-evaluation is reliable enough to trust.

Each of those trade-offs is also a failure mode: an unreliable model votes confidently wrong, a weak verifier gets gamed (reward hacking, Chapter 19, now at inference time), and a miscalibrated self-evaluation steers a search into dead ends. The per-call cost is what bounds them in the end: a method that multiplies token count by tens for a few points of accuracy may not survive the economics of Chapter 76, which is why the field moved toward training the behavior in.

That move is the natural close of this arc. Everything here elicits, at inference, what a fixed model already contains. The next step is to take the traces these methods produce, the verified-correct chains and the high-value search paths, and train on them so the model internalizes the behavior and needs less scaffolding at inference. That step crosses into Chapter 28, and many of its ingredients, sampling many completions, scoring them with a verifier, keeping the good ones, are the same ingredients seen here, now wired into a weight update instead of a decode.

Further reading