Mid-Training: Annealing, Domain Bridges, and Context Extension

The first broad pre-training pass gives a model a general distribution of language, code, facts, styles, and latent procedures. It does not automatically give the post-training pipeline the right starting point. A model can be competent on web text and still be awkward for math traces, code edits, instruction-formatted examples, long documents, or specialist corpora. If post-training is forced to make that jump alone, it must change behavior and repair a distribution gap at the same time.

mid-training names the phase that separates those jobs. It is still pretraining-like: the model usually keeps the same architecture, updates all parameters, and often continues optimizing next-token prediction, the objective of predicting each next token from the ones before it, rather than a preference or reward objective. What changes is the data distribution, the schedule, or the context length. The aim is not to make the model polite, safe, or agentic, but to move its capability distribution closer to the data that later post-training will draw on.

The boundary is operational

The clean definition is not "training that happens in the middle." It is a phase in a sequence. Write a training run as

$$S = \{(D_i, \mathcal{L}_i, n_i)\}_{i=0}^{N},$$

where $D_i$ is the data distribution, $\mathcal{L}_i$ the loss, and $n_i$ the number of steps. The initial phase $i=0$ is broad pre-training. The final phase $i=N$ is post-training or fine-tuning. The intermediate phases $0<i<N$ are mid-training when they still look like pretraining in scale and objective but change the distribution, schedule, or usable context window (Liu et al. 2025).

This makes three boundaries sharp enough to engineer:

• Versus ordinary pre-training. Broad pre-training chooses a general corpus and spends the majority of the token budget on it. Mid-training changes the mixture after the model already has general features.
• Versus continued pretraining. Continued pretraining can switch almost entirely to the target domain. Mid-training usually keeps a general-data component in the mixture, which reduces forgetting while still moving the model toward code, math, instructions, or long documents (Liu et al. 2025; Gururangan et al. 2020).
• Versus post-training. Post-training changes the behavioral interface: supervised fine-tuning (SFT) on demonstrations, direct preference optimization (DPO) from chosen-versus-rejected preferences, reinforcement learning with verifiable rewards (RLVR) from automatically checkable rewards, safety hierarchy, or tool use. Mid-training tries to make the base model a better substrate before those objectives touch it.

Figure 11.1 draws the placement. The important arrow is not "more training." It is "distribution moved before behavior is optimized."

The bridge view

The most useful model of mid-training is a distributional bridge. Suppose $D_0$ is the broad pretraining distribution, where $D_s$ denotes a specialist slice: code, math derivations, instruction-formatted problems, high-quality web, or long documents. A simple mixture schedule is

$$D_t = (1 - \alpha_t)D_0 + \alpha_t D_s,\qquad 0 \leq \alpha_t \leq 1.$$

In words, if $\alpha_t=0$, nothing has changed. If $\alpha_t=1$, the run has become pure continued pretraining on the specialist domain. The middle is the bridge. It keeps enough broad data to preserve general language modeling while making the next post-training distribution less foreign. Controlled experiments in 2025 found this especially helpful for math and code, and found timing more important than the exact mixture weight in the studied code setting (Liu et al. 2025).

This is why a late high-quality slice is not merely a better corpus. It is a timing decision. A clean math or code distribution can be powerful when the model still has enough steps and learning rate left to internalize it. Arrive too late and the slice mostly polishes surface statistics; switch to it too abruptly and it can erase broad capability. There is also the case where the timing is fine but the aim is not: a slice introduced without a downstream target just optimizes the wrong gap.

Three recipes in the wild

Quality annealing gives the final part of pretraining a denser, cleaner signal. OLMo 2 reports a specialized Dolmino Mix 1124 introduced during the annealing phase of pretraining, improving downstream capability while keeping the training data, code, logs, and checkpoints open (OLMo Team 2025). MiniCPM makes the scheduling problem explicit through warmup-stable-decay (WSD): a long stable learning-rate plateau can keep a shared main branch alive, then branch into decay phases for different stopping points or domain-adaptation experiments (Hu et al. 2024). This turns "the end of pretraining" from a single terminal event into a place where recipes can be composed.

Domain mid-training moves the model toward a specialist language before post-training gives it assistant behavior. Qwen2.5-Coder is reported as a code-specific model built on Qwen2.5 and further trained on more than 5.5T tokens with cleaned code, synthetic data, and balanced mixing while retaining general and math ability (Hui et al. 2024). DeepSeekMath starts from a coder base and continues training on 120B math-related tokens mixed with natural language and code before the later SFT and reinforcement-learning stages (Shao et al. 2024). The stack implication is direct: the base model should meet post-training halfway, especially when the target format is far from ordinary web text.

Long-context mid-training changes the length distribution and positional regime. rotary position embedding (RoPE) scaling methods such as Position Interpolation and YaRN make extension cheaper by avoiding raw extrapolation beyond the trained positional range (Chen et al. 2023; Peng et al. 2023). But long context is not only a positional method. Qwen2.5-1M reports long-context pre-training, long-data synthesis, progressive pre-training, and multi-stage SFT to reach 1M-token contexts without giving up short-context performance (Yang et al. 2025). That belongs here because the model is being taught a new sequence-length distribution before serving has to schedule the resulting KV-cache pressure.

What gets tuned

Mid-training has fewer objectives than post-training, but the knobs are dangerous because each one changes what the later model can become.

• Mixture. The data plane from Chapter 6 decides the candidate sources; mid-training decides when and how much of each source enters the stream. Proxy methods such as DoReMi make mixture selection measurable, but the final choice still depends on the target post-training distribution (Xie et al. 2023).
• Timing. The same specialist slice can be weak when introduced late and destructive when introduced as a hard switch. The bridge experiments put timing above exact mixture weight for the code setting they studied (Liu et al. 2025).
• Learning-rate schedule. Cosine decay (the learning rate following a cosine curve down to near zero), WSD, and branch-and-decay recipes decide whether the model can still move. A late high-quality phase with too little learning rate is mainly evaluation dressing; too much learning rate can reopen broad loss.
• Format. Instruction-formatted data can appear before SFT, but it is still next-token data unless the objective changes. This gives the model syntax and discourse forms without yet optimizing preference, helpfulness, or refusal behavior.
• Context length. Long-document mixtures, RoPE scaling, sequence packing, and attention kernels must be aligned with Chapter 10 and Chapter 31, because a longer context learned during training becomes a memory and scheduling cost at inference time.

Failure modes

Mid-training fails quietly. The loss may improve on the new mixture while the model forgets broad language, loses multilingual coverage, or overfits to a format that later post-training will amplify. Specialist code data can move a model toward compiler-friendly syntax at the expense of prose. With math traces, the symbolic style transfers but reliable calculation does not always come with it. Long-context extension is its own trap: a model that passes retrieval probes can still use the middle of its context poorly. And a high-quality web slice can launder benchmark contamination if the decontamination contract from Chapter 6 is not re-run after the slice is assembled.

The engineering failure is to treat mid-training as a cheap epilogue. It touches the optimizer state, learning-rate schedule, data manifest, tokenizer assumptions, position encodings, evaluation decontamination, and checkpoint lineage. It also creates new model variants that need names, model cards, and downstream compatibility tests. If the run cannot answer which distribution changed, when it changed, how much broad data remained, and which evaluations protected against forgetting, the phase is not reproducible.

Where it lands in the stack

Part I now ends with a boundary rather than a cliff. Chapter 5 fixes the budget, Chapter 6 defines the sources, Chapter 7 settles the vocabulary, Chapter 8 and Chapter 9 shape the model body, and Chapter 10 makes the run possible. Mid-training decides whether that run stops as a broad base model or bends toward the capability distribution that Part III will adapt.

That distinction keeps the book's later layers clean. When Chapter 17 adapts behavior, it relies on a code assistant already understanding code syntax, without being the place that taught it. Chapter 21 trains with checkable rewards, yet none of those rewards conjure the mathematical substrate they exploit. Serving long prompts in Chapter 35 inherits one fact it cannot change: whether the model was trained to use long evidence at all. And the buy-tune-continue decision in Chapter 84 turns on this boundary, because it is what tells you whether a problem is really a post-training task or a mid-training one.

Further reading