Reasoning Data and Distillation

Reasoning models are trained not only on questions and answers, but on traces: worked solutions, failed attempts, self-corrections, rejected samples, verifier labels, and long chains later compressed into short ones. That data layer has its own engineering questions: what a reasoning dataset contains, how correct traces are harvested, why small curated sets sometimes work, and how a large reasoner can hand its behavior to a smaller one through distillation, training a small student model on a stronger model's outputs.

A Reasoning Example Is More Than an Answer

A standard supervised example is $(x, y)$: prompt and target. A reasoning example is closer to $(x, z_{1:T}, y, m)$, where $z_{1:T}$ is the trace and $m$ is metadata: verifier result, difficulty, source, language, tool calls, trace length, or process labels. That extra structure changes the training question. The model is not only learning which answer is correct. It is learning a style of search and when to stop.

The simplest loss is still next-token prediction:

$$\mathcal{L}_{\text{trace}}(\theta) = -\sum_{t=1}^{T} \log p_\theta(z_t \mid x, z_{<t}) - \log p_\theta(y \mid x, z_{1:T}).$$

Here the first term teaches the trace tokens, and the second teaches the final answer conditioned on that trace. The data choice decides what this loss teaches. Human-written traces teach a legible style; machine traces teach the teacher's. Filter the set through a checker, and the model sees only the paths that passed it. Long traces teach deliberation, though they teach verbosity too unless something later compresses them. This is why reasoning data is a systems object, not just a file of solutions.

Bootstrapping and Rejection Sampling

STaR was the early form of self-improving reasoning data. Start with a few rationale examples, ask the model to generate rationales for many questions, keep rationales that lead to correct answers, and fine-tune on them. For wrong answers, condition on the correct answer and try to generate a rationale that would have reached it, then repeat (Zelikman et al. 2022). The loop is simple:

def star_round(model, prompts, examples, checker):
    accepted = []
    for x in prompts:
        trace, answer = model.sample_with_rationale(x, examples)
        if checker(x, answer):
            accepted.append((x, trace, answer))
        else:
            trace = model.sample_rationale_given_answer(x, reference_answer(x))
            if checker_trace_or_answer(x, trace):
                accepted.append((x, trace, reference_answer(x)))
    return fine_tune(model, accepted)

Rejection sampling fine-tuning, or RFT, generalizes the filtering idea. Generate many candidate traces, keep the ones whose final answer passes the checker, discard the rest, and train on the accepted set. Yuan et al. found that RFT improves mathematical reasoning, especially when the augmented samples contain diverse correct paths (Yuan et al. 2023). This is the data-side counterpart of self-consistency: sampling raises coverage, the verifier selects, and the selected traces become training data rather than immediate answers.

The failure is selection bias. A verifier that only checks final answers can keep traces with flawed reasoning but correct outputs, or traces that exploit a shortcut. A filtered dataset is therefore not automatically cleaner than a human one. It is only ever clean with respect to the checker.

Small Data, Large Latent Skill

s1 and LIMO made the data question sharper. s1 reports strong reasoning by fine-tuning on only 1,000 carefully curated examples, then using budget forcing at test time (controlling how long the model thinks before it answers) (Muennighoff et al. 2025). LIMO reports strong mathematical reasoning from 817 curated examples and argues that, when the base model already encodes the relevant domain knowledge, a small set of high-quality reasoning templates can elicit the capability (Ye et al. 2025). The implication is not that data no longer matters, only that the marginal value of a reasoning trace depends on what the base model already knows.

This reframes dataset scale. A million low-quality traces may teach format and verbosity. A thousand hard, diverse, correct traces may teach how to activate a latent capability. The useful unit is not the count of examples alone, but the coverage of reasoning moves: decomposition, algebraic transformation, proof planning, code simulation, checking, and stopping. Scale and curation turn out not to be opposites, though: OpenThoughts ran controlled data-recipe ablations up to the 1.2M-trace OpenThoughts3 set and found that scaling a curated pipeline beats tiny hand-picked sets on open base models, its OpenThinker3-7B reaching 53% on AIME 2025 (Guha et al. 2025). That narrows s1 and LIMO to a claim about strong base models, not about data scale in general.

Distilling Long Thought Into Shorter Behavior

DeepSeek-R1 made distillation a central part of the reasoning story. The R1 paper reports that dense smaller models distilled from R1 reasoning traces outperform small models trained with RL directly in that recipe (Guo et al. 2025). Kimi k1.5 reports long2short methods that use long-CoT techniques to improve short-CoT models, giving a production-facing answer to a simple problem: long reasoning is useful, but long answers are expensive (Kimi Team et al. 2025).

Distillation changes the location of the cost. The teacher spends training and test-time compute to find long traces. The student pays a supervised loss to imitate the useful behavior. If it works, production gets part of the reasoning benefit without paying the full token budget on every request. The trade-off is information loss. A short student may learn the result of a deliberation without the ability to reproduce the deliberation under distribution shift.

Weak supervision extends the same idea. Yuan et al. study whether weaker reasoners can supply supervision that improves stronger students, reporting that weak supervision can recover most of the gains of more expensive RL in their setting (Yuan et al. 2025). The result is important because it breaks the assumption that every reasoning dataset must come from a frontier teacher or human expert. The open question is calibration: when does a weak teacher provide useful templates, and when does it teach brittle habits?

Reasoning data is therefore a capital asset. It is expensive to create, hard to audit, and easy to overfit to a checker. The final chapter of this part asks how much of that work should be paid during training, and how much should remain as test-time compute paid per prompt.

Further reading