Synthetic Data and Self-Improvement

Synthetic data is useful because generation is cheap. It is dangerous for the same reason: a model can produce millions of plausible records long before anyone knows whether those records are correct, diverse, or legally usable. The central engineering problem is therefore not generation. It is deciding what evidence makes a generated record safe to train on.

The phrase self-improvement needs similar care. A model does not improve merely because it trains on model-written text. Improvement is a claim about a later model on an independent evaluation. Between those two points sit a task source, a candidate generator, an acceptance signal, an update rule, and a schedule. Any one of them can supply useful pressure or quietly corrupt the loop.

Synthetic data is provenance, not an algorithm

A record is synthetic when a model or simulator generated some material part of it. That description says where the record came from, not how it will be used. The same generated response can become an SFT target, one side of a preference pair, a rollout for reinforcement learning, an evaluation candidate, or nothing at all.

Several research lines now meet under this label. Hinton, Vinyals, and Dean formalized knowledge distillation in 2015 by training a student on a teacher's softened output distribution (Hinton et al. 2015). STaR later selected model-generated rationales whose answers could be checked, then repeated the process (Zelikman et al. 2022). Constitutional AI used model-generated critiques, revisions, and preferences (Bai et al. 2022). Self-Instruct generated instructions, inputs, and responses, filtered invalid or overly similar examples, and fine-tuned on the remainder (Wang et al. 2023). These are related ideas, not stages in one historical ladder.

LIMA supplies an important but narrower result. It showed that 1,000 carefully curated demonstrations could produce strong instruction-following behavior in its setting (Zhou et al. 2023). LIMA did not test a self-training loop. Its evidence supports careful data selection and the value of high-quality demonstrations, not the stronger claim that model-generated data or best-of-$n$ filtering automatically works.

A useful design review separates six questions:

Methods compose across these columns. A teacher can generate candidates that a reward model ranks before a smaller student receives SFT. A current policy can generate both problems and answers while an executor supplies rewards to RL. Calling either pipeline “synthetic training” hides the distinctions that determine its behavior.

Sampling creates opportunity, not correctness

Suppose a prompt $x$ has a set of acceptable responses. Let $q_\theta(y \mid x)$ be the generator's probability of response $y$, and let $z(x,y) \in \{0,1\}$ denote whether an independent oracle would judge that response acceptable. Here the per-sample success probability is

$$p_x = \Pr_{y \sim q_\theta(\cdot \mid x)}[z(x,y)=1].$$

If $n$ candidates are conditionally independent draws with the same $p_x$, their candidate coverage, the probability that at least one is acceptable, is

$$C_n(x) = 1 - (1 - p_x)^n.$$

Here $x$ is the prompt, $y$ is a candidate response, $\theta$ denotes the generator parameters, $q_\theta$ is its response distribution, $z$ is the independent acceptability test, $p_x$ is one-draw success probability, $n$ is the number of draws, and $C_n(x)$ is the resulting coverage. The equation describes opportunity under independent draws. It does not say that a practical selector will find the acceptable candidate, that samples are truly independent, or that more samples add much diversity.

import numpy as np
import matplotlib.pyplot as plt

counts = np.arange(1, 21)
for p in (0.1, 0.3, 0.5):
    coverage = 1 - (1 - p) ** counts
    plt.plot(counts, coverage, marker="o", markersize=3, label=f"p={p}")

plt.xlabel("independent candidates n")
plt.ylabel("candidate coverage C_n")
plt.title("Oracle coverage under independent sampling")
plt.ylim(0, 1.02)
plt.legend()
plt.show()

print("With p=0.10 and n=20, candidate coverage is",
      round(1 - (1 - 0.10) ** 20, 2))

Now replace the oracle with a practical filter $\hat z$. Here $\alpha_x$ is its true-positive rate and $\beta_x$ its false-positive rate on prompt $x$. Its selection precision is

$$\Pr[z=1 \mid \hat z=1,x] = \frac{\alpha_x p_x}{\alpha_x p_x + \beta_x(1-p_x)}.$$

In this expression, $\hat z=1$ means that the filter accepts a response, $\alpha_x=\Pr[\hat z=1\mid z=1,x]$, and $\beta_x=\Pr[\hat z=1\mid z=0,x]$ is the false-positive rate. Even a small $\beta_x$ can pollute the accepted set when truly acceptable samples are rare. False negatives create the opposite problem: they discard good but unfamiliar responses and can narrow the training distribution.

Selection reweights the generator rather than creating correctness from nothing. If $a(x,y) \in [0,1]$ is the probability that the pipeline retains response $y$, the retained distribution is

$$\widetilde q_\theta(y \mid x) = \frac{q_\theta(y \mid x)a(x,y)}{Z_\theta(x)}, \qquad Z_\theta(x)=\mathbb{E}_{y\sim q_\theta(\cdot\mid x)}[a(x,y)].$$

Here $a$ represents all acceptance stages, $Z_\theta(x)$ is the overall acceptance probability for prompt $x$, and $\widetilde q_\theta$ is the normalized distribution of retained responses. More sampling helps only when the generator sometimes reaches useful responses and the acceptance pipeline distinguishes them reliably.

Five patterns that use generated data differently

Teacher distillation

In classical knowledge distillation, a fixed teacher supplies a probability distribution over tokens and a student is trained to match it. In response or sequence distillation, the teacher instead decodes complete text and the student learns those hard targets with ordinary token-level likelihood (Kim and Rush 2016). The second form is common for language models, but it discards most of the teacher's uncertainty.

A teacher is usually the response generator, not a judge applied after generation. Its outputs can transfer formatting, style, domain behavior, and reasoning patterns. They can also transfer refusals, systematic errors, and missing coverage. There is no universal theorem that a student must score below its teacher on every downstream evaluation; students have exceeded their teachers in some distillation settings (Furlanello et al. 2018). The defensible limitation is narrower: teacher-only examples provide no direct corrective evidence for blind spots that useful teacher outputs never cover.

Synthetic tasks and demonstrations

A model can generate the prompts as well as the responses. Self-Instruct begins with seed tasks, asks a model to create new instructions and instances, removes invalid or similar items, and then generates answers for the retained tasks (Wang et al. 2023). This can expand a small seed set cheaply, but the generator still shapes which tasks exist. A pipeline that produces thousands of paraphrases of familiar tasks has increased row count, not coverage.

Task synthesis therefore needs separate checks for prompt novelty, answer quality, difficulty, and relevance to the target population. Prompt generation and response generation should also have distinct provenance. Otherwise an erroneous model-written premise can be mistaken for a hard task with a valid answer.

Filtered self-training

Filtered self-training samples candidates, scores or checks them, and applies SFT to the selected responses. STaR used answer correctness to select generated rationales and then iterated (Zelikman et al. 2022). RAFT samples several candidates per prompt, retains a reward-ranked winner, and fine-tunes on the selected set (Dong et al. 2023). Here best-of-$n$ inference stops after selection; rejection-sampling fine-tuning materializes selected responses as training data and updates the model.

Policy provenance matters here. Candidate collection is on-policy at that moment only if the current learner generated the samples. RAFT can also use a separate generator, and any stored selected set becomes stale as the learner changes. Calling all selected SFT data “on-policy” confuses the collection source with the update algorithm. The selected records are demonstrations, not preference pairs, unless winners and losers are both retained for a preference objective.

AI feedback

An AI critic extends selection to qualities without deterministic checkers, such as tone, helpfulness, or adherence to written principles. Constitutional AI contains two distinct uses. Its supervised phase generates critiques and revisions, then fine-tunes on revised answers. Its RLAIF phase generates AI preferences, trains a preference model, and uses that model as the reward for RL (Bai et al. 2022). It reduced human harmlessness labels in that experiment; it did not remove human-written principles, prompts, or all human helpfulness supervision.

This breadth comes from approximation. A learned judge can have position, verbosity, self-preference, and domain biases. Optimizing harder against a proxy can raise the proxy score while an independent measure falls (Gao et al. 2023). Agreement with the judge used to build the data is therefore not evidence that the resulting model improved.

Verifier-guided learning and self-play

A deterministic checker can provide stronger evidence than a learned critic, but only for the property encoded in the checker. Let $C(x,y) \in \{0,1\}$ mean that response $y$ meets the intended requirement for prompt $x$, and let $V(x,y) \in \{0,1\}$ mean that the implemented verifier accepts it. The verifier is sound when

$$V(x,y)=1 \Longrightarrow C(x,y)=1,$$

Completeness here means

$$C(x,y)=1 \Longrightarrow V(x,y)=1.$$

Soundness prevents accepted wrong answers; completeness prevents valid answers from being discarded. Unit tests, answer extractors, proof statements, and sandboxed executors can all mis-specify the intended requirement. EvalPlus, for example, expanded HumanEval's tests and rejected programs that the original tests had accepted (Liu et al. 2023). A correct final number also does not certify every step of a generated rationale.

A learned process reward model remains a proxy. Let's Verify Step by Step trained outcome and process reward models, including a process model based on 800,000 human step-level labels, and compared them for best-of-$n$ selection. That experiment did not RL-train the generator (Lightman et al. 2024).

RL with verifiable rewards (RLVR) is a different update. It applies automated rewards directly to policy rollouts instead of necessarily storing accepted traces for SFT. DeepSeek-R1-Zero began RL from a pretrained base with no supervised reasoning demonstrations, but it still used curated training questions and rule-based accuracy and format rewards. Full DeepSeek-R1 then combined cold-start data, RL, rejection sampling, SFT, and another RL stage (Guo et al. 2025). It is evidence for verifier-guided RL, not proof that every such system is a generate-filter-SFT flywheel.

Self-play changes who creates the task distribution. Absolute Zero uses one pretrained policy in proposer and solver roles, human-designed task schemas, and a code executor. Its “zero data” setting means zero task data from an external post-training dataset, not zero pretraining data or zero human design (Zhao et al. 2025). R-Zero instead co-trains Challenger and Solver models using majority-vote pseudo-labels. Those labels are not an exact verifier, and the paper reports degradation after repeated rounds (Huang et al. 2025). The two systems should not be treated as equivalent simply because both generate curricula.

Iteration changes the data distribution

One synthetic-data pass and recursive replacement pose different risks. Let $r$ index the training round, $M_r$ be the model at that round, $S_r$ be data generated by $M_r$, and $R$ be retained real or independently verified data. Three regimes are often conflated:

Here $D_{r+1}$ is the next training dataset, $q_{r+1}$ is its sampling distribution, $p_R$ and $p_{S_r}$ are the distributions over real and current-round synthetic data, and $\alpha\in[0,1]$ is the real-data sampling probability. In practice, accumulation also needs explicit weights; a set union does not determine how often each source is sampled.

Shumailov et al. show model collapse under indiscriminate recursive training, with rare parts of the original distribution disappearing before later broad degradation (Shumailov et al. 2024). That result should not be shortened to “synthetic data causes collapse.” Gerstgrasser et al. confirm collapse in a replacement regime but avoid it in their studied accumulation regimes (Gerstgrasser et al. 2024). These are regime-dependent results, not a universal safe mixing ratio. A real-data anchor helps preserve evidence that the current generator cannot recreate, especially rare modes.

Recursive loops have other failure modes that aggregate quality can hide:

Build an auditable loop

The production unit is not “the synthetic dataset.” It is a versioned round with an admission record and an exit decision. A minimal implementation looks like this:

INPUT: frozen task specification, source mixture R, generator M_r, acceptance pipeline A_r
1. Create prompts from approved sources; exclude private evaluation material.
2. Generate candidates with recorded model, template, retrieval, decoding, and seed versions.
3. Deduplicate and decontaminate prompts and responses before quality filtering.
4. Apply deterministic checks, learned critics, and policy rules as separate decisions.
5. Estimate false-positive and false-negative rates on a stratified independent audit.
6. Build D_(r+1) from accepted records plus a deliberate real-data anchor.
7. Train candidate M_(r+1) with the declared update rule.
8. Compare M_(r+1) with M_r on a validation gate that the generator and selector never saw.
9. Promote only if capability, safety, diversity, and contamination gates pass.
10. If any predeclared rollback threshold is crossed, stop the loop and quarantine the round.

In this procedure, $R$, $M_r$, and $D_{r+1}$ have the meanings defined above, while $A_r$ is the complete acceptance pipeline at round $r$. A row-level provenance record should include the source and license, generator checkpoint, prompt template, retrieval corpus, decoding parameters, random seed, parent records, recursive depth, judge or verifier version, threshold, transformations, decontamination results, and acceptance reason. Without that lineage, a later failure cannot be traced or removed.

The independent holdout used as the validation gate must sit outside data construction. It cannot appear in generation prompts, retrieval, judge training, verifier construction, or acceptance-threshold tuning. It may govern promotion and early stopping, so reserve a separate private final test for less frequent claims. The validation evaluator should fail differently from the selector; using the same judge for selection and certification only measures self-consistency.

Track at least four groups of metrics each round:

Predeclare stop and rollback thresholds. If the external score regresses, tail coverage shrinks, or selector disagreement grows beyond its limit, stop the loop, quarantine the new records, revert the checkpoint, and inspect accepted false positives and rejected valid examples before generating another round.

What self-improvement can and cannot establish

Filtered training can make a behavior more reliable when useful responses already appear occasionally. Task generation can expand a curriculum. RL can discover policies that were not present as demonstration text. Distillation can transfer behavior to a different model. None of these observations establishes unlimited autonomous improvement.

The practical question is not whether the data came from a person or a model. It is which parts of the record are synthetic, what independent evidence admitted them, which distribution they represent, and what would stop the next round. A loop that can answer those questions is a data pipeline. One that cannot is only automated self-repetition.

Further reading