Reasoning Data and Distillation

A final answer records whether a model reached a destination. Reasoning data records more of the route: the prompt, a visible trace, the answer, the checks applied to both, and the conditions under which the trace was generated. Those extra fields make it possible to train on successful demonstrations, learn a verifier from labeled failures, form preference pairs, or audit why a data pipeline accepted a sample.

These uses should not be collapsed into one operation. Sampling creates a pool of candidates. Selection changes the distribution of that pool. Supervised fine-tuning imitates selected text. Distillation transfers a teacher's output distribution or selected outputs to a student. None of those steps, by itself, establishes that a written rationale faithfully reports the computation inside either model. This chapter follows the data through each step and states what the resulting model has actually been trained to reproduce.

flowchart TD
    A[Prompt registry and split groups] --> B[Teacher samples candidate traces]
    B --> C[Answer, process, and policy checks]
    C --> D[Accepted demonstrations]
    C --> E[Rejected traces and audit labels]
    D --> F[Deduplicate, balance, and version]
    F --> G[Demonstration corpus]
    E --> M[Version failure labels and audits]
    M --> N[Verifier, preference, or error-model training]
    G --> H[Chosen student objective]
    I[Teacher logits or selected sequences] --> H
    H --> J[Model-size transfer]
    H --> K[Output-length compression]
    J --> L[Held-out quality and cost evaluation]
    K --> L
    N --> L

What a reasoning record must preserve

For training item $i$, $x_i$ denotes the prompt, $z_{i,1:T_i}$ the visible reasoning trace, and $y_{i,1:U_i}$ the final-answer tokens. A trace-trained language model usually masks the prompt and predicts the trace and answer. One explicit per-example loss is

$$\mathcal L_i(\theta) = -\frac{1}{Z_i} \left[ \alpha_z\sum_{t=1}^{T_i} \log p_\theta(z_{i,t}\mid x_i,z_{i,<t}) + \alpha_y\sum_{u=1}^{U_i} \log p_\theta(y_{i,u}\mid x_i,z_i,y_{i,<u}) \right].$$

Here $\theta$ denotes the student parameters; $T_i$ and $U_i$ are the trace and answer lengths; $\alpha_z$ and $\alpha_y$ weight those two regions; and $Z_i$ is the normalization. Setting $Z_i=1$ gives a long completion more total weight because it contributes more token losses. Setting $Z_i=T_i+U_i$ gives every example the same total weight, so each token in a long completion contributes less. Neither choice is neutral. If prompt $i$ has $K_i$ accepted traces, storing every trace as an ordinary row also weights that prompt roughly in proportion to $K_i$. Sampling a prompt first and one of its traces second, or weighting each trace by $1/K_i$, preserves equal prompt weight instead.

The text fields are only part of the record. A useful corpus keeps enough metadata to reproduce admission decisions and detect leakage:

prompt: {text, source_dataset, source_item_id, split_group}
completion: {trace, final_answer, language, token_count, tool_calls}
checks: {answer_result, process_result, policy_result, checker_versions}
generation: {teacher, checkpoint, prompt_template, temperature, seed}
provenance: {license, parent_hashes, generated_at}
disposition: {accepted_for_sft, preference_role, audit_only, rejection_reason}

Each field has a job. split_group keeps paraphrases and shared source problems in one partition. Checker versions make later re-grading possible. Generation settings explain why two nominally identical runs produced different coverage. The disposition prevents a failed trace from silently entering positive supervised data. Metadata that is collected but never used for filtering, sampling, weighting, conditioning, or audit is not part of the training design; it is only storage overhead.

From samples to an accepted distribution

Let $s=(z,y)$ denote a complete trace and answer sampled from a teacher or proposal distribution $q_\phi(s\mid x)$. Let $a(x,s)\in\{0,1\}$ be the admission rule. Rejection sampling does not recover an abstract distribution of correct reasoning. It produces the proposal distribution conditioned on that rule:

$$q_A(s\mid x) = \frac{q_\phi(s\mid x)a(x,s)} {\mathbb E_{s'\sim q_\phi(\cdot\mid x)}[a(x,s')]}.$$

The denominator is the teacher's acceptance probability for prompt $x$ and normalizes the retained samples. The teacher and the admission rule therefore co-author the dataset. If the teacher never proposes a valid path, filtering cannot create one. If the checker accepts a spurious path, supervised training will imitate it.

Sampling more candidates changes which prompts enter the corpus. If one sample passes with probability $p_x$ and the pipeline draws $K$ independent samples, where $p_x$ is the per-sample acceptance probability for prompt $x$, then the probability that the prompt contributes at least one demonstration is

$$P(\text{at least one accepted trace}\mid x)=1-(1-p_x)^K.$$

Here, easy prompts with large $p_x$ enter readily; prompts with tiny $p_x$ may disappear. Keeping every passing sample adds another bias because easy prompts produce more accepted rows. A defensible pipeline records both empty groups and accepted groups, then decides deliberately whether to balance prompts, difficulty bands, domains, or trace families.

Answer, process, and data-utility checks answer different questions:

• Outcome admission asks whether the extracted final answer passes a rule. It does not validate the route to that answer.
• Process admission asks whether steps satisfy human labels, a process reward model, a proof checker, or another stated criterion. A learned process scorer remains an imperfect proxy, as Chapter 27 explains.
• Data-utility admission asks whether the sample adds something useful: a distinct solution structure, an underrepresented difficulty, a permitted license, or a trace length appropriate for the target system.

Rejected traces are useful only when an objective consumes them. They may become negative examples for a verifier, losers in preference pairs, targets for an error classifier, or audit records. Positive-only SFT learns nothing from a rejected row merely because the row was retained in storage.

Bootstrapping traces without confusing answer checks with proof

STaR, the Self-Taught Reasoner, established an influential iterative recipe in 2022 (Zelikman et al. 2022). It starts with a pretrained checkpoint, a small set of rationale demonstrations, and questions with known answers. The reported loop can be written as pseudocode:

M_0 := pretrained checkpoint
M_n := M_0

for round n = 0, 1, ...:
    ordinary := generate rationale + answer with M_n
    keep ordinary samples whose generated answer matches the known answer

    for each failed question:
        rationalized := generate again while revealing the known answer as a hint
        keep it only if the generated answer now matches
        remove the answer hint from the training input

    M_(n+1) := fine-tune a fresh copy of M_0 on all retained samples

The reset to $M_0$ matters: the original algorithm did not simply continue fine-tuning the previous round. The rationalization branch also needs careful interpretation. A trace generated while the answer is visible is a post-hoc explanation conditioned on that answer, not evidence that the model derived the answer independently. STaR filtered on answer correctness, not on a proof of every rationale step. The paper explicitly notes that tasks with high chance-level accuracy can admit poor rationales.

Rejection sampling fine-tuning (RFT) applies the same sample-filter-train pattern without requiring STaR's exact reset and rationalization procedure. In the mathematical-reasoning study that named RFT, an SFT model sampled 100 solutions per GSM8K problem at temperature 0.7, rejected wrong answers and incorrect calculations using Python, extracted ordered equation lists, and selected diverse paths by edit distance (Yuan et al. 2023). In that reported setup, combining rejection samples from multiple models raised LLaMA-7B from 35.9% to 49.3% on GSM8K. The result supports both sampling and structural diversity within that recipe. It does not show that every accepted trace is a valid derivation.

STaR and RFT are offline data construction. Self-consistency instead samples several answers at inference time and aggregates them. Online RL samples from a changing policy and applies gradient updates using rewards. All three may sample multiple completions, but they spend the samples at different stages and optimize different objectives.

Small curated sets are a conditional result

Two 2025 studies showed that reasoning SFT can be data-efficient on an already capable base model. s1 selected 1,000 questions and Gemini-generated traces for difficulty, diversity, and formatting quality, then fine-tuned Qwen2.5-32B-Instruct (Muennighoff et al. 2025). Its final paper reports that the authors' grader judged only 53.6% of the s1K generations correct. The system's headline results also combine the dataset with budget forcing, a decoding-time intervention discussed in Chapter 30. s1 therefore demonstrates a compact, effective recipe; it is not evidence that 1,000 fully correct traces suffice in general.

LIMO used 800 selected mathematical examples with Qwen2.5-32B-Instruct. Those 800 rows were the end of a much larger production pipeline that began with tens of millions of problems and used repeated teacher sampling, heuristics, and manual review. The final COLM paper reports 63.3% on AIME 2024 and 95.6% on MATH-500 and proposes that strong pretrained domain knowledge lowers the number of demonstrations needed to elicit a useful response pattern (Ye et al. 2025). That hypothesis is conditional on the base checkpoint, teacher, prompt distribution, evaluation protocol, and domain. Changing any of them changes the meaning of "800 examples."

Scale is a separate axis. OpenThoughts ran more than 1,000 controlled pipeline experiments and built a 1.2-million-example corpus using QwQ-32B as the teacher. Its ICLR 2026 report compares data recipes while holding the Qwen2.5-7B-Instruct student family fixed and reports 53.3% on AIME 2025 for OpenThinker3-7B (Guha et al. 2026). This is evidence that a curated pipeline can continue to benefit from scale in that matched setup. It is not a controlled refutation of s1 or LIMO because the teachers, students, data, and evaluations differ.

The useful question is therefore not "small or large?" It is how performance changes along independently reported axes: unique prompt families, accepted traces per prompt, teacher and checkpoint, difficulty distribution, verifier quality, duplicate rate, trace length, and total training tokens.

What distillation actually optimizes

Knowledge distillation originally trained a student to match the softened class distribution of a larger teacher (Hinton et al. 2015). For language generation, there are two common targets.

With soft token distillation, the student matches the teacher's next-token distribution at each history $h_t=(x,s_{<t})$, where $x$ is the prompt and $s_{<t}$ is the generated prefix before token $t$:

$$\mathcal L_{\mathrm{soft}} = \tau^2\sum_t D_{\mathrm{KL}} \left(q_T^\tau(\cdot\mid h_t)\,\middle\|\, p_S^\tau(\cdot\mid h_t)\right).$$

Here $q_T^\tau$ and $p_S^\tau$ are teacher and student token distributions softened by temperature $\tau>0$, and $D_{\mathrm{KL}}$ is Kullback-Leibler divergence. This target requires comparable vocabularies and access to teacher logits, which a closed API usually does not provide.

With hard sequence distillation, the teacher generates a completion $\tilde s$, where the symbol denotes one sampled sequence, and the student learns it with ordinary sequence loss:

$$\mathcal L_{\mathrm{hard}} = -\mathbb E_{x,\,\tilde s\sim q_T(\cdot\mid x)} \sum_t \log p_S(\tilde s_t\mid x,\tilde s_{<t}).$$

Teacher samples give a Monte Carlo view of the teacher distribution; greedy or beam outputs emphasize selected modes. If a checker filters the samples first, the student imitates the accepted distribution $q_A$, not the unfiltered teacher. Sequence-level distillation predates reasoning models (Kim and Rush 2016), while rationale-based methods later showed that intermediate text can serve as an additional training target rather than merely a label (Hsieh et al. 2023).

The target can include only the answer, the visible trace plus answer, or separate answer and rationale tasks. These choices teach different output contracts. A trace-trained student is trained to reproduce teacher-like visible text; that is not proof that it recovered the teacher's internal algorithm.

Both targets share a property that is easy to miss. The histories the student is scored on come from the teacher: teacher-forced prefixes in the first case, teacher-sampled sequences in the second. At inference the student conditions on its own text instead, so an early divergence carries it into a region of history space that no training example covered, and every later token compounds the gap. This mismatch between the training distribution and the generation distribution is called exposure bias.

On-policy distillation removes the mismatch by sampling from the student and scoring with the teacher:

$$\mathcal L_{\mathrm{on}} = \mathbb E_{x,\,s\sim p_S(\cdot\mid x)} \sum_t D\!\left(q_T(\cdot\mid x,s_{<t})\,\middle\|\,p_S(\cdot\mid x,s_{<t})\right).$$

The one change from $\mathcal L_{\mathrm{soft}}$ is the expectation. Here $s$ is a sequence the student sampled rather than one the teacher produced, so the student is corrected where it actually goes wrong. $D$ is a divergence between the two next-token distributions, and that choice carries its own meaning. Forward KL asks the student to put mass wherever the teacher does, including modes a smaller student cannot represent; reverse KL asks it to put mass only where the teacher does, which concentrates a limited student on modes it can produce. Agarwal et al. introduced the on-policy form as generalized knowledge distillation and reported reduced error compounding on summarization, translation, and arithmetic reasoning (Agarwal et al. 2024). Gu et al. reached the same sampling scheme from the divergence side, optimizing reverse KL on student samples and reporting better calibration and long-text generation for students from 120M to 13B parameters (Gu et al. 2024).

The price is that the teacher stays resident. Off-policy distillation generates a corpus once and then trains against a file, so the teacher can be a closed API and the training job is ordinary supervised fine-tuning. On-policy distillation needs teacher scores for every student rollout, which puts two models in the training loop and prices the run closer to RL than to SFT. It also needs a shared tokenizer and comparable vocabulary, so an API that returns text but not token distributions cannot serve as an on-policy teacher.

What that price buys is supervision density. A verifiable reward returns one scalar for a whole trajectory (Chapter 28), while the teacher returns a full next-token distribution at every position of the same rollout. That makes on-policy distillation a way to combine capabilities and not only to compress them. MOPD trains separate specialists with RL, then distills all of them into one student on the student's own rollouts, and reports better integration than mixing the specialists' tasks in one run, running them in sequence, fine-tuning off-policy on their outputs, or merging their parameters (Ma et al. 2026). That evidence comes from one industrial post-training pipeline, so it compares recipes under a single setup rather than ranking the four strategies in general.

Smaller models and shorter outputs are different goals

Model-size compression changes parameter count. Output-length compression changes generated tokens. Either may be pursued without the other.

DeepSeek-R1 is evidence for model-size distillation. Its released Qwen and Llama students were fine-tuned on 800,000 samples curated with DeepSeek-R1. For the paper's direct comparison, a distilled Qwen-32B model outperformed one Qwen-32B run trained with more than 10,000 RL steps on the reported reasoning benchmarks (Guo et al. 2025). That is a recipe-specific comparison, not a general theorem that SFT beats RL. The students are smaller than the teacher, but they may still emit long traces. The result does not establish short-output behavior.

Kimi k1.5 studied output-length compression directly (Kimi Team et al. 2025). Its long2short experiments separated four mechanisms:

1. average the weights of long- and short-chain models;
2. sample eight responses and use the shortest correct one for SFT;
3. train with preference pairs that favor the shortest correct response over longer correct or incorrect responses; and
4. run a second RL phase with a length penalty and a shorter rollout limit.

The proprietary, self-reported system also used large SFT stages before these experiments. Within its published comparison, the long2short RL variant gave the best token-efficiency trade-off. This is not ordinary teacher SFT. It is constrained optimization over both correctness and length.

A production evaluation should therefore report at least three quantities together: task quality under a fixed sampling protocol, generated tokens or latency, and model-serving cost. A smaller model that preserves accuracy by producing far more tokens may save memory but not latency. A short model that works only on familiar templates may fail when the prompt requires recovery from an early error.

Weak supervision changes who can be a teacher

A teacher need not be larger than its student. The EACL 2026 weak-to-strong study trained 7B-to-32B Qwen students on traces from weaker 0.5B-to-14B reasoners (Yuan et al. 2026). Those weak teachers had themselves received RL, so the method relocates that cost to smaller teachers rather than eliminating it. In one Qwen2.5-Math pairing, supervision from a 1.5B reasoner recovered 94.34% of the direct-RL gain for a 7B student on MATH. That number is a normalized reasoning-gap result for a particular teacher, student, benchmark, and training recipe, not a universal fraction of RL capability.

The study compared all traces, final-answer-correct traces, and final-answer-incorrect traces. Correct-only filtering usually helped most, but even imperfect traces sometimes improved the student. The authors found that a teacher's structured trace behavior mattered more than parameter count alone. This makes weak supervision a useful design option, while also reinforcing the need to log teacher identity, filtering policy, and benchmark-specific conditions.

Building a defensible reasoning corpus

A production data loop needs stronger boundaries than a benchmark script:

1. Register prompt families before generation. Group originals, paraphrases, translated variants, and shared source documents so they cannot cross train, validation, and test splits.
2. Freeze the generation contract. Version the teacher checkpoint, system prompt, sampling parameters, tools, token budget, and random seed.
3. Store every decision, not only winners. Keep raw candidates, checker outputs, extraction failures, rejection reasons, and later human audits.
4. Separate admission dimensions. Record answer validity, process quality, policy compliance, provenance, and data utility independently before combining them into a final disposition.
5. Deduplicate before splitting and again after generation. Compare source items, normalized answers, trace shingles, code structure, and provenance hashes. A text-only exact match is not enough.
6. Weight deliberately. Report examples, unique prompts, accepted traces per prompt, predicted tokens, and domain mix. These quantities imply different training distributions.
7. Evaluate the student, not the corpus story. Use held-out and adversarial prompt families, matched inference budgets, trace audits, calibration, and general-capability regressions. Compare against answer-only SFT and a compute-matched non-distilled baseline.

The data card should also report licenses and deletion lineage. A generated trace can carry obligations from its source prompt, teacher terms, embedded code, or quoted text. Provenance that cannot be traced backward cannot be reliably removed later.

Payoff and boundary

Reasoning traces can turn expensive sampling into reusable training data and can transfer useful output behavior across models. The asset is not the text alone. It is the versioned relation among prompt, teacher, trace, checks, selection rule, student objective, and held-out result. Remove those relations and a million traces become difficult to audit. Preserve them and the corpus can support SFT, verifier training, preference learning, distillation, and future data rounds without pretending that every accepted explanation is a proof.

Further reading