Mid-Training: Annealing, Domain Bridges, and Context Extension

Broad pre-training ends when its token and compute budget ends. That stopping point does not guarantee that the resulting model is the best starting point for code assistance, mathematical reasoning, a specialist domain, or long documents. Post-training can adapt behavior, but asking a relatively small post-training dataset to repair a large distribution mismatch at the same time makes the transfer harder to diagnose.

mid-training is a name for an intermediate capability-building phase. It usually retains a pretraining-like objective while changing the data mixture, learning-rate schedule, or trained sequence length before final behavior-shaping stages. The practice is older than the name. Domain-adaptive pretraining, for example, was studied explicitly by Gururangan et al. in 2020 (Gururangan et al. 2020). Liu, Neubig, and Xiong proposed a narrower working definition of midtraining in 2025 and tested late mixtures of general and specialist data (Liu et al. 2025). No universal taxonomy has settled yet, so a useful account must describe what a phase actually changes rather than rely on its label.

Name the phase by its role

A training run is an ordered sequence of phases, not an unordered set:

$$\mathcal R=(\mathcal P_0,\ldots,\mathcal P_K), \qquad \mathcal P_i=(\mathcal O_i,\mathcal D_i,\eta_i,B_i,L_i).$$

Here, $\mathcal R$ is the complete run; $\mathcal P_i$ is phase $i$; $\mathcal O_i$ is its training objective; $\mathcal D_i$ is its data distribution; $\eta_i$ is its learning-rate schedule; $B_i$ is its token budget; $L_i$ is its maximum trained sequence length; and $K+1$ is the number of phases. Writing these fields down prevents a vague phrase such as “trained for longer” from hiding the actual intervention.

The phases are best separated by purpose and supervision:

The word typical matters. Mid-training often updates every parameter, but it need not. Next-token prediction is also not a unique boundary because supervised fine-tuning can express a token-level likelihood loss. A phase is classified by the combination of its place in the pipeline, its data and supervision, and the model property it is intended to change.

This also corrects a common false distinction. continued pretraining is a method: resume a pretrained model on more self-supervised data. A mid-training phase can be continued pretraining. In Liu et al.'s experiments, “continued pretraining” names the special comparison that switches to 100 percent specialist data, while “midtraining” keeps specialist and broad data mixed (Liu et al. 2025). Other papers use continued pretraining more broadly, so the mixture must be stated rather than inferred from the name.

Post-training then changes what is supervised. It may use supervised fine-tuning (SFT) on demonstrations, direct preference optimization (DPO) from chosen-versus-rejected preferences, reinforcement learning with verifiable rewards (RLVR) from automatically checked outcomes, or other policy and safety objectives. Those later phases can still teach capabilities. The narrower claim is that a well-chosen bridge can reduce the distribution shift they must absorb.

Quantify the distributional bridge

Let $P$ denote the broad pretraining distribution and $Q$ a specialist distribution such as code, mathematical text, scientific papers, or long documents. At step $t$, a mixture can be written as

$$\mathcal D_t=(1-\alpha_t)P+\alpha_t Q, \qquad 0\leq\alpha_t\leq1.$$

Here, $\mathcal D_t$ is the sampling distribution at step $t$ and $\alpha_t$ is the probability of drawing a token or example from $Q$ under the stated sampler. At $\alpha_t=0$, the phase continues on broad data. At $\alpha_t=1$, it makes a full specialist switch. Values in between replay broad data while increasing specialist exposure. They may reduce forgetting, but do not guarantee it.

The final mixture weight does not reveal the total specialist dose. If the phase contains $T$ equal-token steps, its average specialist share is

$$\bar\alpha=\frac{1}{T}\sum_{t=1}^{T}\alpha_t.$$

Here, $\bar\alpha$ is the fraction of phase tokens expected to come from $Q$; $T$ is the number of equal-token training steps; and $\alpha_t$ is the specialist share at step $t$. A comparison of “early” and “late” introduction is confounded if their $T\bar\alpha$ specialist tokens differ.

For illustration, suppose normalized progress is $u\in[0,1]$, specialist data starts at $s$, and its share ramps linearly to $a$ at the end:

$$\alpha(u)= \begin{cases} 0, & 0\leq u<s,\\ a\dfrac{u-s}{1-s}, & s\leq u\leq1, \end{cases} \qquad \bar\alpha=\frac{a(1-s)}{2}.$$

Here, $s$ is the introduction point, $a$ is the final specialist share, and $\bar\alpha$ is the whole-phase specialist-token fraction under a constant token rate. This linear ramp is an accounting example, not a claim that it is the best schedule. Liu et al. instead studied fixed mixture weights after several introduction points. Their timing and weight effects interacted in small code experiments, so “earlier is better” is not a universal rule (Liu et al. 2025).

The runnable calculation shows why introduction time and final share cannot be reported separately from total exposure.

def specialist_share(progress, start, final_share):
    """Linear mixture ramp over normalized training progress."""
    if not 0 <= progress <= 1:
        raise ValueError("progress must be between 0 and 1")
    if not 0 <= start < 1 or not 0 <= final_share <= 1:
        raise ValueError("invalid schedule")
    if progress < start:
        return 0.0
    return final_share * (progress - start) / (1 - start)


start = 0.60
final_share = 0.30
total_tokens = 100_000_000_000
whole_run_share = 0.5 * final_share * (1 - start)

for progress in (0.0, 0.4, 0.6, 0.8, 1.0):
    print(f"progress={progress:.1f}: specialist={specialist_share(progress, start, final_share):.1%}")

print(f"whole-run specialist share: {whole_run_share:.1%}")
print(f"specialist tokens: {total_tokens * whole_run_share / 1e9:.1f}B")
print(f"broad tokens: {total_tokens * (1 - whole_run_share) / 1e9:.1f}B")

Keep the data schedule separate from the learning rate

“Annealing” is used for two different choices: changing the data near the end of training and decaying the learning rate. They can happen together, but they are not the same intervention. A generic warmup-stable-decay (WSD) schedule is

$$\eta(t)= \begin{cases} \eta_{\max}w(t/T_w), & 0\leq t<T_w,\\ \eta_{\max}, & T_w\leq t<T_s,\\ \eta_{\max}d((t-T_s)/(T_e-T_s)), & T_s\leq t\leq T_e. \end{cases}$$

Here, $\eta(t)$ is the learning rate at step $t$; $\eta_{\max}$ is the stable rate; $T_w$ ends warmup; $T_s$ starts decay; $T_e$ ends the branch; $w$ rises from 0 to 1; and $d$ falls from 1 to the chosen terminal ratio. Linear, cosine, and other choices for $w$ and $d$ produce different schedules.

MiniCPM used WSD so a checkpoint on the stable plateau could support continued training and multiple decay branches (Hu et al. 2024). That is an experimental convenience, not permission to splice schedules casually. A branch must record its source checkpoint, optimizer and scheduler state, new data manifest, learning-rate path, token count, and random seeds. Resetting optimizer state or raising the learning rate is another intervention and needs its own control.

Three uses of the phase

Quality annealing

Quality annealing shifts a late data curriculum toward sources selected for quality or downstream usefulness. In OLMo 2, the first stage consumes roughly 90 to 95 percent of training FLOPs. A final 5 to 10 percent switches to Dolmino Mix 1124 while the learning rate decays. The specialized mix includes filtered web data, decontaminated task data, academic and reference sources, Q&A, and synthetic mathematics (OLMo Team 2025). That result establishes one documented recipe, not a general law that cleaner data is always best late. The source weights, filters, duplicates, and benchmark-overlap checks remain part of the claim.

A “high-quality” bucket should therefore be decomposed into observable properties: source provenance, filter thresholds, document and language mix, deduplication policy, synthetic-data share, and contamination results. Otherwise the phase cannot be reproduced or explained.

Domain mid-training

Domain mid-training uses self-supervised specialist data before behavior adaptation. Qwen2.5-Coder reports 5.2 trillion tokens of file-level training, then roughly 300 billion tokens of repository-level long-context training. Its mixture combines code, text, and mathematics, and its objectives include both next-token prediction and fill-in-the-middle code completion (Hui et al. 2024). DeepSeekMath starts from the pre-decay DeepSeek-Coder-Base-v1.5 7B checkpoint and trains for 500 billion tokens. The reported mixture includes a roughly 120-billion-token filtered math corpus, AlgebraicStack, arXiv, GitHub code, and English and Chinese web text (Shao et al. 2024). These are useful lineage examples, but neither isolates the causal contribution of every mixture component.

Earlier domain-adaptive pretraining work showed that a second self-supervised phase can improve downstream tasks in biomedical, computer-science, news, and review domains (Gururangan et al. 2020). Modern mid-training adds scale, mixture replay, and a deliberate handoff to post-training. DoReMi and related proxy methods can propose domain weights, but proxy gains must still transfer to the target model and target evaluation (Xie et al. 2023).

Long-context mid-training

Long-context mid-training is this book's functional grouping for a phase that changes trained sequence length before final behavior adaptation. It was not part of Liu et al.'s midtraining experiments. A usable extension requires three coordinated changes: a positional method, data containing dependencies at the new lengths, and a training system that can afford longer attention.

For simple Position Interpolation, $L_{\mathrm{train}}$ denotes the original trained length, $L_{\mathrm{ext}}$ the new length, and $m$ a position in the extended sequence. The position supplied to the model is (Chen et al. 2023)

$$s=\frac{L_{\mathrm{ext}}}{L_{\mathrm{train}}}, \qquad m'=\frac{m}{s}.$$

Here, $s$ is the extension factor, $m$ is the token's position in the extended sequence, and $m'$ is the interpolated position supplied to rotary position embedding (RoPE). This keeps positions inside the range seen during the original training. YaRN is not just the same linear map: it treats RoPE frequency bands differently and rescales attention logits, with empirical settings that should not be assumed universal (Peng et al. 2023).

Position handling alone does not teach the model to combine distant evidence. Qwen2.5-1M illustrates the full stack. Its reported pretraining curriculum progressed from 4K through 32K, 65,536, 131,072, and 262,144 tokens. The later one-million-token input claim combines that training with inference-time position and attention methods; its multi-stage SFT belongs to post-training, not to this phase (Yang et al. 2025). A configured input limit is therefore not the same as a trained or effective context length.

The compute budget changes too. For dense causal attention over sequences of lengths $L_i$, the number of query-key pairs is

$$N_{\mathrm{pairs}} =H\sum_i\frac{L_i(L_i+1)}{2}.$$

Here, $H$ is the number of query heads, $L_i$ is the token length of sequence $i$, and $N_{\mathrm{pairs}}$ counts causal attention pairs. At a fixed token budget, replacing many short sequences with fewer long ones raises attention work roughly in proportion to typical sequence length. FlashAttention avoids materializing the full score matrix, but it does not remove this exact dense attention arithmetic. Sequence packing, context parallelism, and memory limits therefore connect this phase directly to Chapter 10.

Design it as a transfer experiment

A mid-training run needs a control matrix, not only a new training loss. At minimum, branch the same source checkpoint into:

1. broad-data continuation for the same token budget;
2. a full specialist switch;
3. one or more mixed schedules; and
4. when post-training compatibility is the goal, a direct-to-post-training branch.

Keep total tokens, batch semantics, evaluation checkpoints, and as much of the optimizer path as possible comparable. Report both the final mixture and cumulative specialist tokens. If a schedule changes several things at once, for example data, learning rate, and sequence length, add ablations or describe the result as a bundle rather than assigning credit to one knob.

Evaluation must remain fixed while the training distribution moves:

• Target transfer: held-out specialist loss and capability evaluations chosen before the run.
• General retention: broad, multilingual, and cross-domain held-outs, plus short-context checks for a length extension.
• Post-training compatibility: the same small SFT or preference recipe applied to comparable intermediate checkpoints. Liu et al.'s strongest claims concern outcomes after SFT at 70M and 160M parameters, not arbitrary frontier-scale models (Liu et al. 2025).
• Long-context use: a matrix of length, evidence depth, distractor count, and task type. Needle retrieval alone is insufficient; models can accept long inputs while failing to use middle evidence or combine multiple facts (Liu et al. 2024; Hsieh et al. 2024).
• Data integrity: re-run deduplication and decontamination on the assembled mixture, complete long documents, and generated instruction-like material.

Training loss on the new mixture cannot serve as the retention metric because the measurement distribution changed. A simple fixed-distribution measure is

$$\Delta_P=J_P(\theta_{\mathrm{after}})-J_P(\theta_{\mathrm{before}}).$$

Here, $J_P$ is loss on a frozen held-out sample from broad distribution $P$; $\theta_{\mathrm{before}}$ is the source checkpoint; and $\theta_{\mathrm{after}}$ is the checkpoint after the phase. Positive $\Delta_P$ means broad held-out loss worsened. It does not by itself prove that every general capability declined, so capability evaluations remain necessary. Every reported loss also needs a named tokenization policy.

Record the handoff contract

The output is a new base checkpoint with lineage. Its handoff record should include:

• source checkpoint and architecture hash;
• optimizer, scheduler, and precision state;
• ordered data manifests, weights, filters, and sampling schedule;
• tokens seen by source, language, format, and sequence-length bucket;
• tokenizer version, packing masks, document boundaries, and position-ID policy;
• trained context length and every RoPE or attention parameter;
• contamination report and fixed evaluation results over time; and
• the post-training recipe versions used to compare transfer.

This record separates a reproducible phase from a model file whose provenance is “continued for a while.” It also lets a later team determine whether a regression entered through data, optimization, length extension, or post-training.

Diagnose the layer before spending the run

Different symptoms point to different interventions:

This diagnosis prevents mid-training from becoming a default answer to every model gap. It is justified when the base checkpoint's data or length distribution is the limiting factor and the expected gain survives a fixed retention and post-training comparison.

Part I ends at that handoff. Chapter 5 set the budget, Chapter 6 defined the sources, Chapter 7 fixed the symbol interface, Chapter 8 and Chapter 9 fixed the model body, and Chapter 10 made the run executable. Mid-training decides whether the resulting base remains broad or moves toward a declared target. Part III then changes the supervision used to shape its behavior.

Further reading