Data Curation and the Training Distribution

Chapter 5 turns a compute budget into a model size and a number of training tokens. It does not say which tokens to use. That choice is the job of data curation.

At fixed architecture, optimizer, and token budget, selection and sampling are major determinants of the model that training produces. They decide which languages, domains, styles, and repeated patterns the model sees. They also decide whether later evaluations measure generalization or accidental exposure to benchmark answers. These effects must be measured for the target model and evaluation suite; no universal quality score can choose the corpus in advance.

The output of curation is therefore more than a directory of text. It is a versioned training distribution: source-specific shards, a sampling policy, exclusion decisions, lineage records, and an decontamination report that states which benchmark-overlap checks were run and what they removed.

The run consumes a distribution, not a folder

Four quantities are easy to confuse:

1. The acquisition pool is everything collected from source snapshots.
2. The retained corpus is what remains after policy, extraction, deduplication, and quality decisions.
3. The mixture assigns sampling probabilities to retained sources.
4. The training stream is the ordered sequence of token instances actually consumed by the run.

A source can occupy 20 percent of stored bytes but receive 5 percent of sampled tokens. A small source can be sampled for several effective epochs, while a large source is only partly read. Corpus inventory is therefore not the same as the distribution learned by the model.

The useful unit also changes by stage. Acquisition works with files, pages, and repository revisions. Deduplication may work with documents, paragraphs, or spans. The model ultimately consumes tokens. Every report should name its unit instead of moving silently between documents, bytes, and tokens.

Build lineage into every stage

The pipeline narrows the acquisition pool, but each rejection also creates evidence. Counts, reason codes, policy versions, and stable record identifiers must travel with the data.

The contract can be stated as an algorithm:

1. Freeze every acquired source as an immutable snapshot.
2. Attach provenance and a policy decision before content filtering.
3. Preserve stable record IDs through extraction and normalization.
4. Apply versioned policy, deduplication, and quality transforms.
5. Compare every retained stage, including synthetic data, with the
   frozen evaluation registry.
6. Write per-source shards, membership manifests, rejection statistics,
   and the sampling schedule.
7. Tokenize and pack with a pinned tokenizer and deterministic ordering.

This work usually runs on CPU-heavy batch infrastructure before the accelerator training job begins. Throughput matters, but resumability matters just as much. A failed stage should restart from immutable inputs without changing which records survive.

Decide whether a source may enter

Publicly reachable, crawlable, licensed, and consented are different properties. Common Crawl supplies timestamped web captures; it does not grant a single license over every captured page. Books, papers, code repositories, licensed collections, and model-generated records need the same source-level review.

A provenance record should include the source URL or repository revision, retrieval time, raw-content digest, creator or publisher when known, stated license and its evidence URL, and dated observations of relevant site terms or crawl preferences. It should also record the project's documented basis for use. Unknown or incompatible cases go to quarantine rather than quietly entering the corpus. Longpre et al. found widespread missing and incorrectly categorized licenses in their audit of more than 1,800 text datasets, which is why a dataset-level label is not enough (Longpre et al. 2023). Legal permissibility remains policy- and jurisdiction-dependent; Chapter 79 develops that boundary.

Personal-information controls are also a process, not a regular expression. Define which classes are excluded, combine rules and model-based detectors, review locale-specific samples, and record detector versions and error rates. No detector proves that a corpus contains no personal information. Flagged content needs restricted access, and a complaint or removal request must be traceable forward to every derived shard and training run. Audits of public corpora have found personal information alongside duplicates, synthetic text, and benchmark material (Elazar et al. 2024).

Extract and normalize without erasing the source

Web captures contain markup, menus, cookie notices, broken character encodings, and boilerplate copied across many pages. Extraction identifies the main text and document boundaries. Normalization then applies a documented Unicode form, repairs chosen extraction artifacts, and standardizes only the features the pipeline has explicitly decided are irrelevant.

The raw and normalized content digests should both remain in lineage. A change in Unicode handling, boilerplate removal, or whitespace rules changes exact identity and can change later deduplication. Rebuilding with a new extractor is a new corpus version, not an invisible implementation update.

Language identification is similarly fallible. A low confidence score may mean mixed-language text, code switching, transliteration, or simply a language the classifier represents poorly. Kreutzer et al. found severe labeling and content quality problems in the multilingual web corpora they audited, especially for lower-resource languages (Kreutzer and others 2022). Report retention by language and review affected samples with relevant language knowledge.

Filtering can also erase legitimate varieties while appearing to remove only undesirable text. Dodge et al. showed that C4's blocklist disproportionately removed text from and about minority groups (Dodge et al. 2021). Rules are easier to inspect than learned scorers, but neither is inherently unbiased.

Deduplicate at the unit that matters

Deduplication serves several different purposes:

• Exact document deduplication hashes canonicalized document bytes and removes identical records.
• Exact span deduplication uses a suffix array or another repeated-substring index to find long copied passages inside otherwise different documents.
• Fuzzy document deduplication compares sets of shingles, such as overlapping word sequences, to find documents with small edits.
• Semantic deduplication uses embeddings to find meaning-level similarity. It can catch rewrites that lexical methods miss, but it also imports the embedding model's definition of similarity (Abbas et al. 2023).

Removing redundancy can reduce memorization and save training steps. Lee et al. found a tenfold reduction in verbatim memorization in their experiments, as well as the same or better accuracy with fewer steps (Lee et al. 2022). The result does not imply that more deduplication is always better. A global policy can remove useful recurring material or leave an unusual remainder. FineWeb, for example, selected per-crawl rather than global fuzzy deduplication after controlled ablations (Penedo et al. 2024).

Jaccard similarity and MinHash

Represent a document $d$ as a set of shingles $S(d)$. The Jaccard similarity of documents $a$ and $b$ is

$$J(a,b)=\frac{|S(a)\cap S(b)|}{|S(a)\cup S(b)|}.$$

Here $S(a)$ and $S(b)$ are the distinct shingle sets, $\cap$ is set intersection, $\cup$ is set union, and $|\cdot|$ counts set elements. The value is zero for disjoint sets and one for identical sets.

The formula requires $n(n-1)/2$ comparisons when applied to every pair of $n$ documents. MinHash, introduced for document resemblance by Broder in 1997, replaces each shingle set with a short randomized signature (Broder 1997). For random permutation $\pi_i$ of the shingle universe, define

$$h_i(S)=\min_{x\in S}\pi_i(x), \qquad \widehat J(a,b)=\frac{1}{m}\sum_{i=1}^{m} \mathbf{1}\!\left[h_i(S(a))=h_i(S(b))\right].$$

The terms have the following meanings: $h_i(S)$ is the minimum permuted rank in set $S$ under permutation $i$; $x$ is a shingle; $m$ is the signature length; $\mathbf{1}[\cdot]$ is one when its condition is true and zero otherwise; and $\widehat J$ is the estimated Jaccard similarity. For ideal independent permutations, $\mathbb{E}[\widehat J]=J$ and $\operatorname{Var}(\widehat J)=J(1-J)/m$. More signature entries reduce variance, although one realized estimate need not improve monotonically.

locality-sensitive hashing (LSH) (LSH) splits each signature into $b$ bands of $r$ rows, so $m=br$. If two documents have Jaccard similarity $s$, the idealized probability that at least one band matches is

$$P_{\mathrm{candidate}}(s)=1-\left(1-s^r\right)^b.$$

The symbols mean the following: $P_{\mathrm{candidate}}(s)$ is the probability of becoming a candidate pair; $s\in[0,1]$ is true Jaccard similarity; $b$ is the number of bands; and $r$ is the rows per band. More bands at fixed signature length increase candidate recall and false positives. More rows per band make the gate stricter and can increase false negatives. LSH produces candidates, not proof of duplication; an implementation may verify candidates with exact Jaccard or another pinned rule before forming duplicate clusters.

The cell below uses a shared random ordering of the union of two sets. That is equivalent to drawing a random permutation for MinHash. The sets have exact Jaccard similarity $0.6$; the estimate is the fraction of permutations whose minimum element agrees.

from math import sqrt
from random import Random

rng = Random(18)
universe = list(range(5000))

# Two 800-element sets with 600 shared elements: J = 600 / 1000 = 0.6.
A = set(rng.sample(universe, 800))
outside_A = [x for x in universe if x not in A]
B = set(rng.sample(sorted(A), 600) + rng.sample(outside_A, 200))

true_j = len(A & B) / len(A | B)
elements = sorted(A | B)
max_perm = 1024
matches = []

for _ in range(max_perm):
    order = elements.copy()
    rng.shuffle(order)  # one shared random permutation of the union
    min_a = next(x for x in order if x in A)
    min_b = next(x for x in order if x in B)
    matches.append(min_a == min_b)

for m in [16, 64, 256, 1024]:
    estimate = sum(matches[:m]) / m
    standard_error = sqrt(true_j * (1 - true_j) / m)
    print(
        f"m={m:>4}  estimate={estimate:.3f}  "
        f"true={true_j:.3f}  standard error={standard_error:.3f}"
    )

Figure 6.3 separates the all-pairs baseline from one favorable indexed case. The terms in its bound are signature construction, which costs $O(nm)$, and the $K$ candidate pairs emitted by the index, giving roughly $O(nm+K)$ work. It is close to linear only when candidates per document stay bounded. In a degenerate bucket, $K$ can still approach $n(n-1)/2$.

Duplicate clusters must also be resolved deterministically. Sort records by a stable identity, build all verified similarity edges, compute connected components, and choose one representative by a pinned policy. Dropping a document merely because it collided in an LSH bucket can delete distinct text; choosing the first record seen makes the corpus depend on partition order.

Quality is a measured property

There is no context-free label called “high-quality text.” A filter can favor fluent prose while harming code, favor Wikipedia-like language while removing conversation, or improve one evaluation while narrowing another domain.

Two broad tool families are common:

• Rules measure length, repetition, symbol ratios, boilerplate patterns, language confidence, or other inspectable features.
• Learned scorers estimate similarity to a reference set or predict a label such as educational value.

Neither family is inherently high-recall, high-precision, or unbiased until the target is defined and checked against reviewed data. CCNet's reference-model perplexity score, for example, measures similarity to its Wikipedia-derived reference distribution, not an objective property called fluency (Wenzek et al. 2020). Report retention and error rates by relevant language, dialect, geography, source, and topic slices. Inspect excluded records as well as accepted ones.

The strongest validation is a fixed-budget training ablation. Hold the raw pool, tokenizer, model, optimizer, token budget, and evaluation suite constant; change one curation decision; then compare loss and downstream slices. FineWeb publicly ablated extraction, filtering, and deduplication choices. DataComp-LM later provided a 240-trillion-token raw pool and standardized model training to compare data recipes, finding model-based filtering important in its baseline setting (Penedo et al. 2024; Li and others 2024). These are controlled results, not universal rankings of datasets.

Figure 6.4 shows the intended mechanism as a thought experiment. A curated stream helps only if it yields more useful signal for the target distribution per token consumed. Filtering that removes useful coverage can move the curve the other way.

Turn retained sources into sampled tokens

Suppose the retained corpus has $K$ sources. Each static mixture defines

$$P_{\mathrm{train}}(x)=\sum_{k=1}^{K}w_kP_k(x), \qquad w_k\geq 0, \qquad \sum_{k=1}^{K}w_k=1.$$

The symbols mean the following: $P_{\mathrm{train}}(x)$ is the probability of sampling token sequence $x$ for training; $P_k(x)$ is the sampling distribution within source $k$; $w_k$ is that source's mixture weight; and $K$ is the number of sources. With a total budget of $D$ sampled tokens, source $k$ contributes an expected $D_k=w_kD$ tokens.

If source $k$ contains $U_k$ retained token instances, the following formula defines its expected effective epochs:

$$e_k=\frac{D_k}{U_k}=\frac{w_kD}{U_k}.$$

The terms mean the following: $e_k$ is the expected number of full passes over source $k$; $D_k$ is its sampled-token allocation; and $U_k$ is its unique retained-token count. This quantity exposes accidental oversampling that a table of mixture percentages can hide.

Weights can come from expert judgment or compute-matched proxy experiments. DoReMi first trains a small reference model on initial domain weights, then trains a same-size proxy with group distributionally robust optimization on per-domain excess loss. It averages the evolving weights and uses them to resample data for a larger model (Xie et al. 2023). The reported transfer from a 280M proxy to an 8B model is evidence for that experiment, not a guarantee across tokenizers, domain definitions, model families, or scales.

A static mixture holds each $w_k$ fixed. A curriculum uses time-dependent weights $w_k(t)$ or changes example ordering as training progresses. Late up-weighting of a high-quality or domain-specific slice is one form of curriculum, often called data annealing or a cool-down mixture when paired with the final learning-rate phase. Its interaction with the training schedule is covered in Chapter 11.

Treat synthetic data as a derived source

Synthetic data can rephrase existing text, generate textbook-style examples, distill a stronger model, or create verified exercises. TinyStories showed that a narrow synthetic story distribution could teach coherent English to models below ten million parameters (Eldan and Li 2023). The phi-1 experiments reported strong code-generation results from a 1.3B model trained on a curated mixture containing synthetic textbook and exercise data (Gunasekar et al. 2023). WRAP mixed real C4 text with model-generated rephrasings and improved efficiency in its tested setting (Maini et al. 2024).

These results do not make generated tokens automatically cheap, factual, novel, or useful. Generation consumes inference compute; verification adds cost; and a generator can reproduce benchmark content, source errors, or a narrow style. Recursive replacement of original data by outputs from successive model generations is the high-risk collapse setting studied by Shumailov et al.; synthetic augmentation with preserved original data is a different regime (Shumailov et al. 2024).

Every generated record should retain its parent source IDs, generator provider and checkpoint, prompt or template version, decoding settings, timestamp, verifier result, and applicable usage terms. Evaluate factual fidelity, diversity, source coverage, contamination, and downstream performance before assigning a synthetic mixture weight.

Decontamination protects an evaluation claim

Deduplication asks whether training records repeat one another. Decontamination asks whether training data contains protected evaluation material. The second question needs a frozen registry of benchmark versions before curation begins.

Whole-document Jaccard is a poor detector when a short benchmark item is copied into a long page. For evaluation item $e$ and training document $d$, the following formula defines an evaluation-side $n$-gram containment score:

$$C_n(e,d)=\frac{|G_n(e)\cap G_n(d)|}{|G_n(e)|}.$$

The symbols mean the following: $G_n(z)$ is the set of normalized token $n$-grams in text $z$; $n$ is the chosen sequence length; $C_n(e,d)$ is the fraction of the evaluation item's $n$-grams found in the training document. A policy may flag a pair when $C_n(e,d)\geq\tau$, where $\tau$ is a benchmark-specific threshold. The normalization, $n$, common-gram suppression, retrieval method, threshold, and adjudication rule all belong in the report.

Lexical matching has a known boundary. Paraphrases, translations, code rewrites, and synthetic restatements can preserve the answer while sharing few surface strings. Yang et al. demonstrated such evasions and found overlap in synthetic datasets (Yang et al. 2023). Semantic or model-assisted review can increase coverage, but it introduces its own errors. A decontamination report proves what was checked and removed; it cannot prove that no leakage exists.

The hand-off should contain benchmark and corpus version hashes, detector and normalization versions, thresholds, reviewed samples, removal counts by source and benchmark, removed token totals, and before-and-after corpus manifests. Use opaque record IDs for restricted material rather than copying private data or benchmark answers into a broadly visible report.

Make every accepted record traceable

Reproducibility does not require pretending that a live URL will never change. Freeze raw snapshots and accepted shards as immutable, content-addressed objects. Rebuilding from those inputs with pinned code should reproduce the recorded membership and shard checksums. Re-fetching a live source creates a new corpus version.

A machine-readable manifest should contain at least:

corpus:
  id: <stable corpus name>
  version: <semantic version>
  owner: <responsible team>
  intended_uses: [...]
  excluded_uses: [...]
source_record:
  uri_or_revision: <WARC record, URL, or repository commit>
  retrieved_at: <timestamp>
  raw_digest: <content hash>
  origin: <human, model-generated, or unknown>
  rights_evidence: <license, terms, consent, and dated evidence>
pipeline:
  code_commit: <revision>
  environment_digest: <container and dependency digest>
  transforms: <models, thresholds, seeds, and configurations>
  stage_counts: <accepted and rejected counts by reason>
lineage:
  parent_record_ids: [...]
  dedup_cluster_id: <stable cluster>
  retained_representative: <stable record ID>
evaluation:
  registry_digest: <protected benchmark versions>
  detector_policy: <normalization, matching, thresholds, review>
output:
  tokenizer_digest: <pinned tokenizer>
  shard_digests: [...]
  sampling_policy: <weights and schedule>
  consumed_by_runs: [...]
  tombstones: <removals and affected descendants>

The public datasheet can summarize purpose, sources, transforms, known limitations, and slice statistics. Sensitive raw records, personal information, and protected benchmark material remain access-controlled. Dolma illustrates the value of publishing both corpus documentation and curation tooling (Soldaini et al. 2024); Datasheets for Datasets provides a broader documentation framework (Gebru et al. 2021).

Validate the corpus before the full run

A corpus earns a training budget through evidence at several scales:

1. Stage accounting. Reconcile document, byte, and token counts before and after every transform. Investigate unexpected retention jumps.
2. Reviewed samples. Sample accepted and rejected records by source, language, score band, and policy reason. Estimate errors with uncertainty.
3. Distribution comparisons. Measure which domains, languages, hosts, licenses, and document types each stage adds or removes.
4. Duplicate audits. Report exact, span, and fuzzy cluster distributions, including the largest clusters and representative policy.
5. Contamination audits. Run the frozen registry against every path into the corpus, including synthetic and imported datasets.
6. Sampler tests. Generate a dry-run stream and verify realized weights, effective epochs, ordering, packing, and shard balance.
7. Compute-matched ablations. Train small models with one data decision changed at a time, then compare held-out loss and target slices.
8. Replay and removal. Trace sampled records backward to sources and a source forward to all descendants and consuming runs.

Before committing the full run, record the acquisition snapshots, policy basis, retained membership, filter audits, mixture weights, schedule, effective epochs, decontamination limitations, tokenizer version, and output shard hashes. That record makes the learned distribution inspectable and gives later evaluation a defensible account of what the model could have seen.

Further reading