Eliciting Reasoning

Changing the weights is not the only way to change a model's answer. With the weights fixed, a system can ask for intermediate work, draw several candidates, extract their answers, compare them, or search among partial solutions. These choices spend more computation after the request arrives. They are therefore inference procedures, not new capabilities installed by training.

That distinction keeps the claims in this chapter modest. A longer trace can help later tokens reach a better answer, but it can also carry an early mistake forward. More samples can increase the chance that a good candidate exists, but they do not identify it. Search can recover from a poor branch, but only if its evaluator recognizes the better one. The useful unit is the complete inference pipeline, not the instruction “think step by step.”

Fixed weights, variable inference

Elicitation changes the computation around a model while its weights stay fixed. A production procedure has at least five choices:

These choices compose. A single chain uses one prompt, one candidate, and no comparison. Self-consistency samples several chains and selects an answer mode. Best-of-$n$ scores complete candidates. A tree repeatedly generates and scores partial states. Calling all of them “reasoning prompts” hides where the additional work and the new failure modes enter.

For one generated rationale $r$ and final answer $y$, an autoregressive model can be written as

$$p_\theta(r,y \mid x) = p_\theta(r \mid x)\,p_\theta(y \mid x,r).$$

Here $x$ is the prepared prompt, $\theta$ is the fixed parameter set, $r$ is generated intermediate text, and $y$ is the final answer. Because $r$ precedes $y$, it becomes part of the context used to generate $y$. This factorization explains how intermediate text can affect an answer. It does not establish that $r$ faithfully reports all computation that caused the answer, or that a longer $r$ must be better.

One trace: chain of thought and decomposition

Intermediate-text computation predates the name chain of thought. Scratchpad work made models emit intermediate calculations before an answer, while Wei et al. showed that a few worked chain-of-thought examples could improve arithmetic, commonsense, and symbolic reasoning in sufficiently large models (Nye et al. 2021; Wei et al. 2022). The size qualification matters: the reported gains were not uniform across smaller models, and the experiments established benchmark improvements, not a general law about all multi-step tasks.

Kojima et al. removed the worked examples and used the instruction “Let's think step by step” (Kojima et al. 2022). Their Zero-shot-CoT procedure was a two-stage prompt: the first call generated intermediate text, and a second, answer-specific prompt extracted the final answer in the required format. “Zero-shot” referred to the absence of task-specific exemplars in that prompt. It did not mean the model had never seen reasoning demonstrations during training.

Both results show that changing the requested output sequence can change performance without a weight update. That does not prove that the capability was latent and merely unlocked. Generated rationale tokens alter the context and therefore the subsequent computation. Whether this helps is task and model dependent, and filler tokens alone did not reproduce the chain-of-thought gains in the ablations reported by Wei et al.

A chain of thought (CoT), a written chain of intermediate reasoning steps, leaves the model responsible for deciding what the next useful step is. least-to-most makes that organization explicit. It first asks for a decomposition, then solves the resulting subproblems in dependency order while carrying earlier answers forward (Zhou et al. 2023). This can help when the decomposition is reliable and each earlier result supplies what the next subproblem needs. It can also fail before the first solution step: a missing subproblem, wrong dependency order, or early wrong answer contaminates everything downstream. The paper's strongest extrapolation result came from a task-engineered SCAN setup; its gains on other tasks were smaller. Decomposition is a design choice, not a universal upgrade.

Several traces: self-consistency is mode estimation

A single sampled chain can be unrepresentative. self-consistency draws several chains at nonzero temperature, extracts the final answer from each, and returns the most frequent normalized answer (Wang et al. 2023). This is an empirical answer mode, often called plurality voting. It is not necessarily a strict majority.

Let $r_i$ be sampled trace $i$, let $e(r_i)$ be the answer extractor, and let $\mathcal{Y}$ be the set of normalized answers. With $n$ samples,

$$\widehat{\pi}_n(y \mid x) = \frac{1}{n}\sum_{i=1}^{n}\mathbf{1}[e(r_i)=y], \qquad \widehat y_{\text{mode}} = \arg\max_{y\in\mathcal{Y}}\widehat{\pi}_n(y \mid x).$$

Here $\mathbf{1}[\cdot]$ is one when its condition holds and zero otherwise, $\widehat{\pi}_n$ is the observed answer frequency, and $\widehat y_{\text{mode}}$ is the selected answer. A mode can win with less than half the votes. The implementation must also define answer extraction, equivalence, normalization, abstentions, and tie handling. For example, 0.5, 1/2, and “one half” may be the same answer even though their strings differ.

The familiar claim that voting amplifies accuracy needs stronger assumptions. In a binary toy model, suppose each chain is conditionally independent and correct with probability $p$, and let the odd sample count be $n=2m+1$. Majority-vote accuracy is

$$A_n(p)=\sum_{k=m+1}^{n}\binom{n}{k}p^k(1-p)^{n-k}.$$

Here $m=(n-1)/2$, $k$ is the number of correct votes, and $\binom{n}{k}$ counts which $k$ samples are correct. Under these assumptions, voting improves with $n$ when $p > 1/2$ and makes the result worse when $p < 1/2$. Real traces have many possible wrong answers and correlated errors. The correct answer can win as the unique population mode even below 50 percent, or a shared wrong answer can become increasingly dominant.

The following calculation adds a simple common-mode error. Here $\rho\in[0,1]$ represents the probability that all chains share one outcome; otherwise they are independent. Increasing $\rho$ preserves one-chain accuracy but removes most of the benefit from voting.

from math import comb

def independent_majority(n, p):
    threshold = n // 2 + 1
    return sum(
        comb(n, k) * p**k * (1 - p)**(n - k)
        for k in range(threshold, n + 1)
    )

def shared_error_majority(n, p, rho):
    independent = independent_majority(n, p)
    return rho * p + (1 - rho) * independent

p = 0.55
for rho in (0.0, 0.25, 0.75):
    values = [shared_error_majority(n, p, rho) for n in (1, 3, 5, 11, 21)]
    print(f"rho={rho:.2f}: " + ", ".join(f"{value:.3f}" for value in values))

Self-consistency works best when sampling produces meaningfully different routes, the answer extractor is reliable, and systematic errors do not form the dominant answer cluster. Diversity of wording is not enough if every trace repeats the same mistaken premise.

From traces to search

Sampling complete traces spends the same amount of attention on every candidate. Search allocates the next unit of compute after inspecting partial work. Tree of Thoughts made this controller explicit for language-model inference (Yao et al. 2023). A concrete search must define five task-specific elements:

Tree of Thoughts instantiated these choices with task-designed thought units, model generation and self-evaluation, and breadth-first or depth-first search. Its large Game of 24 gain is evidence for that setup on a task with useful branches. It is not evidence that trees dominate sampling on arbitrary workloads. Xie et al. similarly combined a prompted self-evaluation score with stochastic beam search and reported task-dependent gains (Xie et al. 2023). In both cases the evaluator is part of the method, not an oracle. Wider search can amplify evaluator error by presenting more high-scoring mistakes.

AlphaZero-style language search belongs on the boundary of elicitation. The cited TS-LLM work trained policy, value, and outcome-reward components from rollout data, then used them during search (Wan et al. 2024). Its inference policy may be frozen during a run, but the guidance did not appear without training. The next chapter, Chapter 25, develops the search layer in detail and explains why the clean board-game analogy breaks for free-form language states.

Selection: a good candidate must exist and be recognized

Best-of-$n$ samples $n$ complete candidates, scores them, and returns the candidate with the highest score. Two events must go right. First, the candidate set needs coverage: at least one acceptable answer must exist. Second, the selector must recognize one.

Here $c_i$ denotes candidate $i$, $z(c_i)\in\{0,1\}$ is its independent acceptability judgment, and $j=S(c_1,\ldots,c_n)$ is the index chosen by selector $S$. Define per-request selection regret as

$$R_{\text{sel}} = \max_{1\le i\le n} z(c_i) - z(c_j).$$

Here $R_{\text{sel}}=1$ means an acceptable candidate existed but the selector missed it. If every candidate is unacceptable, coverage failed and selection regret is zero even though the system also fails. This separation prevents a common diagnostic mistake: adding samples addresses candidate coverage, while improving a checker or scorer addresses selection.

Selectors provide different kinds of evidence:

• An answer mode measures agreement after answer extraction. It needs no separate judge, but a correlated mistake can become the winning answer.
• A deterministic checker executes an encoded criterion. Exact numeric equality checks a final value, unit tests cover only their cases, and a proof kernel checks a formal statement rather than the model's translation from natural language.
• A learned scorer estimates outcome or process quality from training data. It can make both false acceptance and false rejection errors, especially on the high-scoring tail produced by aggressive search.
• Human review can cover open-ended criteria, but it adds latency, cost, and disagreement.

Cobbe et al. applied a learned outcome verifier to GSM8K (Cobbe et al. 2021). They generated candidate solutions, labeled them by whether the final answer matched, trained a verifier on those labels, and ranked 100 candidates at test time. The labels did not establish that each intermediate step was valid, so this was outcome supervision rather than process supervision. Verification improved their results and scaled better with more training problems than their fine-tuning baseline, but performance eventually declined when still larger candidate sets exposed high-scoring mistakes. Controlled experiments on reward-model optimization show the same general risk: optimizing a learned proxy more strongly can eventually reduce the gold score (Gao et al. 2023).

The broader empirical record reinforces the distinction. Repeated sampling can keep raising oracle candidate coverage while majority and learned-score selection flatten (Brown et al. 2024). That is a workload-specific finding, not a universal sample-count threshold. The amount of useful inference compute depends jointly on the generator, the candidate distribution, the selector, and the task.

Put the budget and failure policy in the design

No method should be the default merely because it spends more tokens. Route by what the task exposes and what the service can afford:

A production controller needs a latency budget as well as token and scorer budgets. Independent samples can run in parallel when capacity exists, reducing wall-clock delay but not total work. It also needs a fallback for timeouts, parser failures, ties, an empty accepted set, and evaluator disagreement. Returning the direct answer, abstaining, or escalating to a human are different product decisions and should be explicit.

The minimum telemetry for each request is the model and prompt version, sampling settings, generated-token count, candidate count, extracted answers, selector type and version, scores or checker outcomes, chosen candidate, wall-clock latency, and fallback path. Offline evaluation should then separate:

• single-candidate accuracy;
• oracle candidate coverage;
• selected-answer accuracy;
• selection regret when labels are available;
• answer diversity and pairwise agreement;
• false acceptance and false rejection by evaluator version;
• cost per correct answer, plus median and tail latency.

Increase a budget only while its marginal gain survives on a selection-independent holdout. Stop or roll back when coverage rises but selected accuracy does not, when score tails drift away from audited quality, or when latency and cost exceed the service-level objective. Training may amortize recurring behavior, but it does not make inference free; elicitation may instead be routed only to hard or high-value requests. The decision is a workload calculation, not a universal contest between prompting and training.

Elicitation therefore changes context and decoding without updating weights. A single trace changes the sequence the model conditions on. Multiple traces change candidate coverage. A selector changes which candidate becomes the answer. Search changes where the next unit of compute goes. Keeping those effects separate makes the next chapters easier: Chapter 25 develops the controller, Chapter 27 develops the evaluator, and Chapter 30 turns both into a budget-allocation problem.

Further reading