Data Curation and Quality

Chapter 5 fixed the compute budget; this chapter asks what fills it. At fixed compute and architecture, the corpus is what separates a strong model from a mediocre one. The path from raw web bytes to token-id shards decides which distribution the model will learn, which capabilities will be over- or under-fed, and whether later evaluation numbers can be trusted at all.

A corpus is a system, not a script

A pre-training run consumes a fixed compute budget set by Chapter 5. That budget buys a token count, and the corpus that fills it is the single biggest lever on final quality and the one least transferable from open recipes. The difficulty is that the failures here are silent. A duplicated shard, a filter that strips a dialect, a benchmark question that leaked through a synthetic-data step: none of these throw an error, and all of them surface only at evaluation, where they are expensive to diagnose and often impossible to undo without re-running. So the corpus must be built as a system whose every stage is auditable and whose hand-offs to the rest of the stack are explicit contracts.

Two of those hand-offs bind the rest of the book. The vocabulary is consumed by the architecture in Chapter 7 (this chapter treats the tokenizer only as a pointer; its design lives there). The decontamination report, the audit showing which benchmark overlaps were removed from training data, is consumed by evaluation in Chapter 47, which supplies the eval registry this chapter checks against.

This is not just a scripting problem. Everything below runs at petabyte volume as distributed batch or stream jobs (Apache Spark, Ray, or custom MapReduce-style clusters) across large CPU fleets, separate from and ahead of the GPU training cluster. The hard parts are throughput, determinism, and resumability, not the per-document algorithms. A re-run must produce byte-identical shards, or the mixture and decontamination contracts become unverifiable.

With those contracts in view, the next sections follow a byte from the open web through the narrowing filters that decide whether it survives, then step back to the three decisions that are locked early, the contract that protects the numbers, and the histories that explain why the field still argues about all of it.

A byte's journey through the narrowing filters

The pipeline is a sequence of narrowing filters, each cheap relative to the training run it feeds, arranged so the cheapest cuts come first.

Where the byte comes from

The bulk substrate is the web. CommonCrawl is the raw input: WARC/WET extraction, boilerplate stripping, language identification, and URL/host-level filtering. CommonCrawl is raw and noisy, and the value is in the pipeline applied to it, not the crawl itself. Around that web core sit curated sources, books, papers, encyclopedic and reference text, source code with license and quality signals, and non-English text at deliberate ratios. Each source has different licensing, quality, and dedup characteristics, and each competes for the same token budget.

Before any filtering, the bytes are made uniform. Cleaning and normalization covers Unicode normalization, repair of text extraction artifacts, PII handling, and toxicity and safety pre-filtering. The goal is to remove garbage without scrubbing legitimate distributional variety: the registers, dialects, and domains that a model needs to see to generalize.

The first cut: deduplication

Deduplication runs three mechanisms at three granularities. Exact hashing removes identical documents. Suffix-array or substring methods remove near-duplicate spans inside otherwise distinct documents. MinHash with locality-sensitive hashing (LSH) does fuzzy document-level dedup at corpus scale: it estimates Jaccard similarity between documents (the fraction of pieces two documents share) from a small set of hashed shingles (the overlapping k-word pieces of a document), so the cost is sub-quadratic rather than all-pairs. Dedup recovers wasted compute and reduces verbatim memorization. The standing tension is aggressiveness against losing genuinely distinct content.

Dedup is not merely a cost optimization, and the evidence says so. Lee et al. showed that deduplicating training data makes models better, not just cheaper: it cuts memorization and improves held-out loss (error measured on data the model was not trained on) (Lee et al. 2022). SemDeDup pushed the idea into embedding space, removing semantic near-duplicates that surface-form hashing misses (Abbas et al. 2023). The through-line is that redundancy is not neutral; it actively degrades the model, so the filter that removes it is a quality filter in disguise.

Figure 6.4 shows why MinHash with LSH avoids the all-pairs blow-up: each document collapses to a short signature, LSH bands route similar signatures into shared buckets, and a full Jaccard estimate is computed only for the few documents that already collide in a bucket.

Run this to see how a MinHash signature estimates Jaccard similarity from a handful of hashes, and how the estimate sharpens as num_perm grows.

import numpy as np

rng = np.random.default_rng(0)
universe = 5000                       # vocabulary of shingles
A = set(rng.choice(universe, 800, replace=False).tolist())
B = set(A) ^ set(rng.choice(universe, 400, replace=False).tolist())
true_j = len(A & B) / len(A | B)      # exact Jaccard

def minhash_estimate(A, B, num_perm):
    seeds = rng.integers(1, 2**31 - 1, size=num_perm)
    a = np.array(sorted(A)); b = np.array(sorted(B))
    hits = 0
    for s in seeds:                   # min of a hashed permutation = MinHash
        if (a * s % universe).min() == (b * s % universe).min():
            hits += 1
    return hits / num_perm            # P(min collides) = Jaccard

for num_perm in [16, 64, 256, 1024]:
    est = minhash_estimate(A, B, num_perm)
    print(f"num_perm={num_perm:>5}  est={est:.3f}  true={true_j:.3f}")

Figure 6.3 makes the cost argument quantitative. Naive all-pairs comparison grows as the square of the corpus size, which is hopeless at petabyte scale, while LSH keeps the candidate count close to linear by only ever comparing documents that already share a bucket.

The second cut: quality filtering

What survives dedup still has to clear a quality bar, and two families set it. Heuristic gates are rule-based: length, symbol-to-word ratio, repetition, stop-word presence, and a perplexity threshold from a reference model (how surprised that model is by the text, with low perplexity meaning fluent). They are cheap, auditable, and high-recall for obvious junk, but brittle at the margins. Classifier filters are learned: a model scores each document against a reference of "good" text, for example a classifier trained to distinguish curated text from raw web. Classifiers have higher precision but carry a structural risk, they encode the reference set's notion of quality into the entire corpus. The reference perplexity filter traces directly to CCNet, which paired language identification with a perplexity score from a reference language model (Wenzek et al. 2020).

The decisions locked early

By the time the byte clears quality filtering, the per-document work is done. What remains is global: how to combine sources, how to order them, and whether to manufacture more. These decisions are cheap to state and expensive to change, because each one constrains the others.

Mixture is the ratios across web, code, curated, and multilingual, plus per-source weights. It can be hand-tuned, scaled from small-proxy ablations (controlled runs that change one thing to measure its effect), or optimized directly. DoReMi frames mixture selection as group distributionally robust optimization: train a small proxy, use it to find weights that minimize worst-case excess loss across domains, then apply those weights to the full run (Xie et al. 2023). The mixture is locked early and expensive to revisit, because changing it means re-deriving the curriculum and often re-running proxy ablations.

Curriculum is the ordering and phasing of data over training: easy-to-hard, domain phasing, or saving a high-quality slice for late. This is the ordering within the data plane. It is distinct from the annealing phase, which up-weights high-quality data near the end of training and belongs to mid-training, covered in Chapter 11.

Synthetic data is model-generated and templated text: rephrasing or augmenting web text, textbook-style generation, and distillation targets. The principle is that a small amount of dense, clean, on-distribution text can be worth far more than its token count in raw web. It is powerful for filling capability gaps, and Chapter 23 treats it as a first-class post-training tool. The risks are distributional collapse, factual drift, and laundering eval content back into training. Because the last risk feeds directly into the contract that follows, synthetic data is also where the pipeline most needs the next stage to be strict.

Decontamination as a contract

Decontamination is N-gram or substring overlap detection against the eval registry, an overlap threshold, a drop-or-flag decision, and a report emitted as the contract to evaluation. This is the stage that earns the chapter its place in the stack. The report, not a shared implementation, is the hand-off.

The threshold is not a free parameter. N-gram overlap detection is itself evadable: rephrased benchmark samples slip past surface-level matching, which is exactly why the threshold and method are a negotiated contract rather than a private default (Yang et al. 2023). A loose threshold leaks benchmarks and inflates scores. A strict one over-removes legitimate near-duplicates and shrinks the corpus. Either way the decision belongs to the boundary between this chapter and Chapter 47, not inside the data team.

How the recipe got here: two histories

From assembled corpora to pipeline-as-product

Early corpora were assembled, not engineered. The Pile curated 800GB from twenty-two diverse sources and made the components explicit (Gao et al. 2020). C4, built for T5, showed that a handful of heuristic cleaning rules over CommonCrawl already moved downstream quality (Raffel et al. 2020). The pivot to pipeline-as-product came with CCNet, which combined language identification with a perplexity filter from a reference language model (Wenzek et al. 2020), and then with RefinedWeb, whose claim was sharper: web data alone, filtered and deduplicated hard enough, can match or beat curated corpora (Penedo et al. 2023). FineWeb carried this further with public ablations that fixed compute and architecture and varied only the data, turning corpus construction into a measurable science rather than folklore (Penedo et al. 2024).

The synthetic turn

The synthetic turn arrived through two results. TinyStories showed that a narrow, fully synthetic corpus could teach small models coherent English (Eldan and Li 2023). The phi line, opened by "Textbooks Are All You Need," argued that textbook-quality synthetic and filtered data could reach strong reasoning at a fraction of the usual token count (Gunasekar et al. 2023). Rephrasing the Web (WRAP) made the cheaper version of the bet: rephrase existing web text into cleaner styles rather than generating from scratch, gaining compute and data efficiency without inventing content (Maini et al. 2024). The newest and least-documented source is agent transcripts, tool-use traces and verified multi-step trajectories recycled as training signal, foreshadowed by datasets like ToolBench (Qin et al. 2023). The open questions there are provenance, dedup against the source model's own outputs, and contamination from embedded eval tasks.

Figure 6.5 traces why the choice is structural, not a single tunable ratio: each camp branches into a different stack, cost structure, and contamination risk.

What you trade, and what breaks

Every stage above is a dial, and turning it the wrong way fails silently. The dials are worth naming together because they interact.

• Quantity versus quality. More tokens help only if they are not junk or duplicates. Past a point, aggressive filtering and dedup beat raw volume, as sketched in Figure 6.6: held-out loss on raw web flattens once added tokens mostly repeat content the model has already seen, while the same compute on deduplicated and filtered tokens keeps falling. The hard part is knowing where that point is for a given compute budget, and that point moves with the budget set in Chapter 5.

• Dedup aggressiveness. Stronger fuzzy dedup removes memorization risk and wasted compute, but can delete legitimately rare-but-distinct content. The threshold is a recall-precision dial with no universally right setting.
• Heuristic versus classifier filtering. Heuristics are cheap, auditable, and bias-light but coarse. Classifiers are precise but import the biases of their reference set into the entire corpus, stripping registers, dialects, or domains wholesale.
• Synthetic share. Synthetic data fills gaps cheaply but risks collapse, factual drift, and eval laundering. The ratio trades diversity against control.
• Decontamination strictness. A loose threshold leaks benchmarks and inflates scores. A strict one over-removes legitimate near-duplicates and shrinks the corpus. The threshold is a contract term, not an internal detail.

The failure modes are worth naming because each bites at a different time. Contamination leaks inflate results and are detected late, if at all. The dedup dial cuts both ways: too loose, it wastes compute and raises memorization risk; too tight, it quietly removes rare content. Broader still is filter bias amplification, which strips legitimate variety across the whole corpus, while synthetic collapse or drift narrows the distribution or propagates the generator's errors. Worst of all is a mixture mistake: it under- or over-trains a capability, and because the mixture is locked early and surfaces only at evaluation, it is the costliest to diagnose and redo.

The determinism discipline

After the design choices comes the operational requirement: determinism. Because mixture and decontamination are contracts, every stage must be reproducible to the byte. In practice that means content-addressed inputs, pinned filter thresholds and model versions, and a manifest that records, for each output shard, the exact transforms that produced it. A representative dedup step is fuzzy document matching by MinHash and LSH, sketched here against its method rather than any one codebase:

# Fuzzy dedup: bucket by LSH, then drop near-duplicates within a bucket.
def dedup(docs, threshold=0.8, num_perm=128):
    seen = LSHIndex(threshold=threshold, num_perm=num_perm)
    for doc in docs:
        sig = minhash(shingles(doc.text), num_perm=num_perm)
        if not seen.query(sig):      # no near-duplicate already kept
            seen.insert(doc.id, sig)
            yield doc                 # else: drop, counted in the manifest

The decontamination step is the same shape with a different index: build n-gram signatures for every item in the eval registry, then drop or flag any training document whose overlap exceeds the contract threshold, and write the count and identities of removed documents into the report that evaluation consumes.

Further reading