Tokenization

Chapter 6 ended with token-id shards. Tokenization defines those ids: the map between text and the integers a model actually reads. This early map decides how expensive each language is to process, where code and numbers get split, and how much of a document's sequence length is spent encoding it. It is also why one preprocessing choice becomes one of the hardest decisions in the stack to revisit. And so byte-level BPE appears here as the dominant compromise, with Unigram and tokenizer-free schemes set beside it as different answers to the same constraints rather than cosmetic alternatives.

A model never sees text

A transformer does not see text. It sees a sequence of integers drawn from a fixed vocabulary, each mapped to a row of an embedding matrix, the lookup table mapping each token id to a learned vector, one row per vocabulary entry. Something has to turn a stream of Unicode into that sequence, and the tokenizer is that something. The choice looks like plumbing and behaves like a foundation.

The reason is a conjunction of three constraints, each harmless alone and unforgiving together.

The vocabulary must be finite and fixed before training begins, yet text is open. New words, code, emoji, and languages the designer never enumerated must still encode without failure. A table of words can never be complete, so completeness has to come from somewhere other than the table.

The segmentation must be cheap. Tokenization runs over the entire corpus on CPU fleets ahead of the GPU run, as covered in Chapter 6, so an elegant but slow scheme is not affordable at corpus scale.

And the result is permanent. The embedding and output matrices are sized to the vocabulary (the output matrices, or output projection, convert the model's final vectors back into a score per vocabulary entry), so the vocabulary cannot change without retraining the model. A tokenizer trained on a skewed corpus caps quality for everything under-represented in it, and that ceiling is invisible until evaluation.

Together, these constraints make tokenizer choice both quality-critical and close to irreversible. Everything that follows is an attempt to satisfy the first two constraints while living with the third.

The subword bargain

The dominant answer is subword tokenization: a vocabulary of pieces smaller than words but larger than characters, sized so common words are single tokens while rare words decompose into known parts. This is a bargain struck directly against the open-vocabulary constraint. Any string is representable because the pieces bottom out at units that always exist, and frequent text stays short because common sequences are merged into single ids.

Byte-pair encoding (BPE) is the workhorse construction. Start from a base alphabet, count adjacent pair frequencies across the corpus, merge the most frequent pair into a new symbol, and repeat until the vocabulary reaches its target size (Sennrich et al. 2015). The learned merge list is the tokenizer; encoding replays the merges in order.

# BPE training, conceptual
vocab = set(base_symbols)
merges = []
while len(vocab) < target_size:
    pair = most_frequent_adjacent_pair(corpus, vocab)
    merges.append(pair)
    vocab.add(merge(pair))

A worked example makes the loop concrete. Figure 7.2 runs two merges on a tiny corpus of the word low repeated, where the most frequent adjacent pair is absorbed into a new symbol each round until the common sequence is a single token.

Run the pseudocode for real: this trains BPE on a tiny corpus, prints the learned merges, then shows how common and rare words split into a different number of tokens.

from collections import Counter

def merge_pair(word, a, b):
    out, i = [], 0
    while i < len(word):
        if i < len(word) - 1 and word[i] == a and word[i + 1] == b:
            out.append(a + b); i += 2
        else:
            out.append(word[i]); i += 1
    return out

corpus = "low low low lower lowest newest wider".split()
words = [list(w) + ["_"] for w in corpus]
merges = []
for _ in range(6):
    pairs = Counter()
    for w in words:
        for a, b in zip(w, w[1:]):
            pairs[(a, b)] += 1
    if not pairs:
        break
    (a, b), _ = max(pairs.items(), key=lambda kv: (kv[1], kv[0]))
    merges.append((a, b))
    words = [merge_pair(w, a, b) for w in words]

print("learned merges:", [a + "+" + b for a, b in merges])
for w, toks in zip(corpus, words):
    print(f"{w:7s} -> {' '.join(toks)}  ({len(toks)} tokens)")

The decisive refinement is to run BPE over raw bytes rather than Unicode characters. With 256 byte values as the base alphabet, every possible input encodes with zero out-of-vocabulary risk, even text the tokenizer never saw, and there is no separate unknown token to handle. This is byte-level BPE, and it is why the same tokenizer can ingest English, code, a new emoji, and a script absent from its training data. Byte-fallback gives a near-equivalent guarantee in character-based schemes by decomposing any unseen character into its underlying bytes. The open-vocabulary constraint, which sank every word-level table, is now satisfied by construction rather than by hoping the table was large enough.

There is a second question the bargain leaves open: not what the pieces are, but how to choose a split when several are possible. BPE is greedy and frequency-driven, with no probabilistic model of how a string should split. The unigram language model offers an alternative: posit that a sentence is a bag of subword pieces with independent probabilities, start from a large candidate vocabulary, and prune pieces whose removal least hurts the corpus likelihood (Kudo 2018). Unigram yields a principled segmentation and, because it defines a distribution over segmentations, supports subword regularization: sampling alternative splits at training time as a form of augmentation (Kudo 2018).

Two parameters set by the data pipeline govern the rest. Vocabulary size trades sequence length against matrix size, and the tokenizer's own training corpus bakes in which languages and domains get short, efficient encodings. Both are frozen here and consumed, not chosen, downstream.

From words to bytes

The construction above did not arrive whole. Tokenization moved from words to bytes in three steps, each removing a failure of the last. Figure 7.3 traces the lineage, with the specific failure each stage carries and the one its successor removes.

Word-level vocabularies came first and broke on the open-vocabulary problem: any word absent from the table became a single unknown token, discarding its content. Character-level models removed the unknown token but made sequences long and forced the model to relearn spelling from scratch, spending depth on what a vocabulary could have stored.

Sennrich et al. (2015) brought BPE, invented for data compression, to neural machine translation as the subword compromise, and it became the default (Sennrich et al. 2015). The unigram language model, introduced by Kudo (2018), offered a probabilistic alternative with subword regularization (Kudo 2018). Kudo and Richardson (2018) then packaged both in SentencePiece, which treats input as a raw stream including whitespace, so the tokenizer is language-independent and needs no pre-tokenization rules (Kudo and Richardson 2018). The final step was byte-level BPE, which made the base alphabet the 256 byte values and closed the out-of-vocabulary gap entirely. Byte-level BPE is the construction behind most current frontier tokenizers, with unigram and SentencePiece remaining common in multilingual and non-Latin-script settings.

The open frontier is whether to tokenize at all. tokenizer-free schemes operate directly on bytes or characters, learning the segmentation inside the model rather than fixing it in a preprocessing step. They remove the frozen vocabulary and its biases, at the cost of longer sequences and the compute to process them. This is a live research direction, not a settled replacement.

What you cannot take back

The history above reads like progress, and it is, but every choice it settled became a setting that someone has to make once and then live with. Each tokenizer setting is a balance with a knee, and several of them set a ceiling that only shows up later. Figure 7.4 makes the central one visible: a larger vocabulary shortens sequences but enlarges the matrices sized to it, so the two costs cross rather than agree.

• Vocabulary size. A larger vocabulary shortens sequences, which makes training and inference cheaper per document and improves multilingual fertility, the number of characters carried per token. It also enlarges the embedding and output matrices and can starve rare tokens of the occurrences they need to train a good embedding. The optimum is a function of corpus and budget, not a constant.
• Training corpus for the tokenizer. Train it on the final data mixture or on a separate curated set? The choice permanently bakes the language and domain balance into the vocabulary. A mismatch with the real mixture spends vocabulary on the wrong things and leaves under-represented languages with long, expensive encodings.
• BPE versus unigram. BPE is greedy, fast, and ubiquitous, with strong tooling and broad compatibility. Unigram is probabilistic, supports subword regularization, and often segments multilingual text more evenly, at the cost of a heavier training procedure and less ecosystem inertia behind it.
• Multilingual fairness. Petrov et al. (2023) show that a tokenizer tuned on English-heavy data splits other languages into many more tokens for the same meaning (Petrov et al. 2023). That inflates cost and latency for those users and shortens their effective context window, a structural unfairness fixed in the vocabulary itself. This is the trade-off that most directly caps downstream quality, and it is set entirely here. Figure 7.5 shows the shape of the gap: the same sentence costs several times more tokens in some languages than in English.

The reason these matter more than ordinary hyperparameters is lock-in. A vocabulary trained on a skewed mixture caps multilingual quality and inflates token cost for under-served languages, and it cannot be changed without retraining the model, because the embedding and output matrices are sized to it. Vocabulary extension during continued training can graft on new tokens as a partial workaround, but it does not undo the original bias. In short, the tokenizer is decided once and lived with for the model's life, which is why a choice that looks like preprocessing deserves the scrutiny of an architecture decision.

Knobs you set once

Naming the lock-in is not the same as dismissing the work. In practice a tokenizer is trained once, with a handful of decisions that are expensive to get wrong, and two recurring knobs deserve their own attention beyond vocabulary size and training corpus.

The first is digit handling: splitting numbers into individual digits so arithmetic does not depend on whether "1234" happened to merge into one token. The second is whitespace handling, where SentencePiece-style schemes encode spaces as ordinary symbols so detokenization is exact and reversible (Kudo and Richardson 2018). The build itself is the BPE training, conceptual loop above: count, merge, repeat, then freeze the merge list and vocabulary as the artifact every later stage consumes.

That artifact is the whole point. It is small, it is produced early, and it binds everything downstream to the language and domain balance present in the corpus that made it. The tokenizer arrives looking like a preprocessing step and turns out to be the first irreversible commitment in the model's life, and that is the lens worth carrying into the architecture that consumes it.

Further reading