Synthetic Data and Self-Improvement

Human-written training data is finite, and the best of it is the most expensive. When fresh human labels stop scaling, the training signal must come from somewhere else: a stronger model, the model's own filtered outputs, an AI critic, or a verifier. Distillation, rejection sampling, self-play (the model generates its own training problems and solutions, then learns from them), and AI feedback are four answers to one question. Iterating any of them creates a data flywheel, and each loop has its own ceiling.

The upstream question

Post-training needs labeled examples: prompts paired with good responses, or responses paired with a judgment of which is better. Chapter 18 describes the policy and rubric layer that gives those judgments meaning. The supervised and preference methods that consume those labels live in Chapter 17 and Chapter 19; Chapter 20 covers the direct-preference family, and Chapter 21 separates checkable rewards from learned ones. The upstream problem sits before all of them. Human demonstrations and human preference labels are slow, costly, and bounded by the skill of the annotator. A frontier model is often already better than the median labeler at the task you want to teach, so paying humans to produce its training data caps the result at human-median quality and exhausts the budget long before the data does.

So one question sits under everything that follows: who decides a response is good enough to train on? If the answer can be something cheaper than a human, post-training scales with compute instead of with annotation headcount, and the data stops being the wall that Chapter 6 describes for pre-training. The catch is that a self-produced signal is only as trustworthy as whatever filters it, and a bad filter teaches the model its own mistakes.

There turn out to be four answers, and they form a single lineage: a steady retreat from the human labeler, removing the human from one role at a time. The rest follows that retreat, source of judgment by source of judgment, and each step both buys something and gives something up.

One loop, four answers

Before the four answers, the shape they share. Every method here is the same loop: generate candidates, filter them, train on what survives, and possibly repeat. The methods differ in exactly one place, the source of judgment that drives the filter, as Figure 23.2 shows.

When that loop is run repeatedly, each round's improved model generating the next round's data, it is a data flywheel: the model's current capability becomes the engine that produces the data for its next increment. The flywheel is not a fifth method. It is what you get by iterating any of the four.

Filtering is the engine under every answer, and it works for a plain reason: a model that cannot reliably produce a correct answer on the first try can often produce one somewhere in $n$ tries, and a cheap filter can find it. Training on the filtered winners moves probability mass onto behavior the model was already capable of but did not reliably emit. This is the same elicitation framing that explains why a few thousand curated examples move behavior so much: the capability is latent, and the data selects for it rather than installing it. LIMA is the clean evidence, showing that a small, carefully filtered set outperforms a large noisy one (Zhou et al. 2023).

Vary the per-try success rate $p$ and watch how fast pass@$n$, the chance at least one of $n$ samples is correct, climbs toward one as $n$ grows.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)
ns = np.arange(1, 21)
for p in (0.1, 0.3, 0.5):                # per-try success probability
    analytic = 1 - (1 - p) ** ns         # pass@n if tries are independent
    sims = [(rng.random((3000, n)) < p).any(axis=1).mean() for n in ns]
    line, = plt.plot(ns, analytic, label=f"p={p}")
    plt.scatter(ns, sims, s=10, color=line.get_color())
plt.xlabel("samples n"); plt.ylabel("pass@n (at least one correct)")
plt.legend(); plt.title("Why filtering works: best-of-n success"); plt.show()
print("p=0.10: a model right 10% of the time succeeds within 20 tries",
      round(1 - 0.9 ** 20, 2), "of the time")

Figure 23.3 lays the four sources out as one stepwise withdrawal, each step removing a human from a different role. The sections below follow it in order.

Answer one: borrow a teacher

The first move was to borrow a teacher. A more capable model generates the responses, and a weaker student trains on them. The judgment is implicit: the teacher's output is taken as the target. This is response distillation, learning from the teacher's text, not its logits (the raw scores it assigns to every possible next token), and once a strong aligned model existed, it was the cheapest way to align a weaker one: have the strong model write the instruction-following data. This made capable open models possible without each lab running its own human-preference pipeline, and it remains the default way open models are aligned.

Its limit is structural. A student trained only on a teacher's distilled outputs rarely exceeds the teacher on those skills, and it inherits the teacher's biases and refusals along with its competence. Distillation is off-policy: cheaper and more stable than learning from the model's own samples, but it drifts from the live model and caps at the teacher. The data is borrowed, and so is the ceiling it imposes.

Answer two: filter your own outputs

The next move was to filter your own outputs, so the model could improve without a stronger teacher to copy. Sample $n$ responses from the current model, score them, keep the best, and fine-tune on the winners. The judging now comes from a scorer applied to the model's own samples. Iterated, this is rejection-sampling fine-tuning: on-policy preference data without a full RL loop. reward ranked fine-tuning (RAFT) formalized it, ranking sampled responses by a reward and fine-tuning on the top of each group (Dong et al. 2023).

The gain is that the signal is on-policy, drawn from the model itself, which closes the gap between what the model trains on and what it actually generates. The model stops imitating a foreign distribution and starts selecting from its own. The cost is that the ceiling is now set by the scorer rather than by a teacher, which moves the problem rather than removing it: the loop is only as good as whatever scores the samples.

Answer three: replace the labeler

The third move reached past the demonstrator to replace the labeler. A model, prompted with written principles, produces the preference signal that a human annotator would otherwise provide. Constitutional AI is the canonical form: an AI critiques and revises responses against a short written constitution, and the resulting pairs train the model, cutting the human cost of the preference signal down to the principles themselves (Bai et al. 2022). The judge here is a model reading rules. This is reinforcement learning from AI feedback (RLAIF); the RL machinery that consumes that preference signal is Chapter 19, but the signal's origin is here.

The reach is what makes it attractive: an AI critic can score open-ended responses, helpfulness and style and tone, where no ground-truth checker exists. The cost is that the judge is a proxy. Push on it and the model finds outputs the critic rates highly and humans do not. Replacing the labeler removes the human from the labeling role but installs an exploitable stand-in where the human used to be.

Answer four: improve against a checker

Most recently, the move has been to improve against a checker. Generate many candidate solutions, run each through a checker, a unit-test suite, an answer checker, a proof checker, and keep the ones that pass. The judgment is ground truth, not a proxy, which makes this the cleanest loop, because the filter cannot be manipulated by style or confidence. When the filter is a verifier rather than a learned scorer, the loop can bootstrap on math and code: generate solutions, keep the verified-correct traces, train on them, and repeat.

DeepSeek-R1 is the open data point, including an R1-Zero variant that bootstraps reasoning from a base model with no supervised cold start (Guo et al. 2025). The reinforcement-learning mechanics that turn verified traces into a reasoning model belong to Chapter 28; what belongs here is the narrower observation that a sound verifier turns the model's own correct outputs into a renewable training set. Where the verifier is a learned process reward model (PRM) rather than an exact checker, the signal can be made finer-grained by scoring each reasoning step rather than only the final answer (Lightman et al. 2023), though a learned scorer reintroduces the proxy problem the exact checker had escaped.

The reason this loop bootstraps where the others cap is that the ceiling rises with the model. The model can be pushed to produce correct outputs it could not reliably produce before, and the verified-correct set grows with each round. That is the cleanest version of the data flywheel: capability generating the data for its own next increment, with a filter that does not accept a wrong answer.

With a sound executor as the verifier, even the prompts can come from the model. Absolute Zero has a single policy propose its own tasks and then solve them, with a code executor checking both roles, and trains from no external data at all (Zhao et al. 2025); R-Zero co-evolves a Challenger that invents problems and a Solver that answers them, again from zero external data (Huang et al. 2025). This is the self-play the opening glossed, taken to its limit: the human is gone from the demonstrations, the labels, and now the questions, and the verifier is the only input left to bound the bootstrap.

How far does the bootstrap go?

Walking the four answers leaves one tension unresolved, and it is the real question. Can self-improvement create genuinely new capability, or is it bounded by whatever does the judging?

The answer to how far it goes is decided one layer down, by whether the domain is verifiable at all. That is the constraint that shapes everything above it.

The four answers therefore fall along a single antidiagonal between two ends: cheap-and-broad coverage that buys an exploitable proxy, and trustworthy-but-narrow ground truth that costs coverage. A learned reward model or an AI critic can score any response, including open-ended ones, but it is a proxy. A verifier cannot be manipulated by style or confidence, but it only exists where ground truth is checkable, and its coverage is the new attack surface. Figure 23.4 places the four sources on that axis.

Two more trade-offs cut across the antidiagonal. On-policy filtering (rejection sampling, self-play) trains on what the model actually generates and closes the train-serve gap, where distilling a teacher is off-policy, cheaper and more stable but drifting from the live model. And a single generate-filter-train pass is simple and bounded, where iterating compounds gains but also compounds errors: a small bias in the filter, applied every round, becomes a learned habit, and the model can collapse onto whatever the filter rewards (model collapse: quality degrades as it trains on its own outputs (Shumailov et al. 2024)). The flywheel needs a filter sound enough to survive repetition. Distillation has one further cost the others do not: it may violate the teacher's terms of service, and from-scratch alignment, expensive as it is, remains the only path to surpassing the available teachers.

The one line that decides everything

The minimal loop is short enough to state directly, and writing it out shows why the four answers are one method. Rejection-sampling fine-tuning, the RAFT recipe (Dong et al. 2023), makes the loop concrete:

# Rejection-sampling fine-tuning, one round (sketch).
data = []
for prompt in prompts:
    cands = [model.sample(prompt) for _ in range(n)]
    scored = [(judge(prompt, c), c) for c in cands]
    best = max(scored)[1]            # judge: reward model, AI critic, or verifier
    if keep(best):                   # threshold or verified-correct
        data.append((prompt, best))
model = finetune(model, data)        # iterate: next round samples from this model

The single line that determines everything is judge. Swap in a stronger model and the loop is distillation; a reward model or AI critic and it is rejection sampling or AI feedback; a unit-test or answer checker and it is verifier-based self-improvement. The surrounding machinery is identical; the trust you can place in the result is entirely a property of judge. The four answers of this chapter are four bindings of one variable.

Two operational realities follow from that. First, the filter's false negatives starve the loop: a verifier that rejects correct-but-unusual answers, or a scorer miscalibrated against good outputs, removes exactly the data you wanted. Second, the loop's failures are quiet. A flywheel running on a filter that can be exploited shows rising filtered-set quality by the filter's own measure while held-out quality stalls or regresses, the same Goodhart signature that reward over-optimization produces in Chapter 19. The defense is to keep an independent evaluation outside the loop, which is the job of Chapter 47, and to prefer a verifier wherever the domain admits one.

Further reading