Scaling Laws and Compute Allocation

Training a base model requires several choices that become expensive to revise once the run begins: the architecture, parameter count, data mixture, number of training tokens, optimizer, and schedule. Scaling laws address one part of that planning problem. For a fixed model family and training setup, they estimate how held-out loss changes with model size, data, and compute.

That is narrower than predicting whether a model will be useful. A scaling law can help choose the size of a training run and forecast its validation loss. It does not choose the data, guarantee a downstream capability, or account for serving cost unless those factors are added to the objective. Keeping those boundaries visible is the key to using the method well.

What the forecast measures

Language-model scaling studies usually measure next-token prediction on held-out text. Given a sequence $x_1,\ldots,x_T$, a model with parameters $\theta$ assigns a probability to each token after seeing the preceding tokens. Its average cross-entropy loss is

$$\begin{aligned} \ell_t&=-\log p_\theta(x_t\mid x_{<t}),\\ L_{\mathrm{eval}}(\theta)&=\frac{1}{T}\sum_{t=1}^{T}\ell_t. \end{aligned}$$

Here $T$ is the number of evaluated token positions, $x_t$ is the true token at position $t$, $x_{<t}$ is its preceding context, and $p_\theta(x_t\mid x_{<t})$ is the probability assigned by the model. The value $\ell_t$ is the negative log-probability at position $t$. This is next-token prediction. The average negative log-probability is cross-entropy: it measures the model's surprise on a fixed held-out distribution. With natural logarithms, the unit is nats per token and perplexity is $\exp(L_{\mathrm{eval}})$; with base-2 logarithms, the unit is bits per token and perplexity is $2^{L_{\mathrm{eval}}}$.

Lower held-out loss means better probabilistic prediction on that evaluation distribution. Comparisons are meaningful only when the tokenizer, held-out data, and loss convention are held fixed. A small loss improvement may or may not produce a useful change on a particular task.

The word token is used in two different counts. A vocabulary contains token types, while a training run processes token instances. In particular, training tokens are not vocabulary entries, and repeated passes over the same text count again. If a corpus contains $U$ unique token instances and is read for $R$ full epochs, the run processes approximately $D=RU$ training tokens.

From small runs to a loss surface

Across a matched family of models, held-out loss often decreases smoothly as the parameter count $N$ and training-token count $D$ increase. A commonly used fit is

$$L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta}.$$

$E$ is a fitted asymptote for the chosen tokenizer and data distribution. The terms $A/N^\alpha$ and $B/D^\beta$ estimate the excess loss caused by limited model capacity and limited training data. The positive constants $A$, $B$, $\alpha$, and $\beta$, together with $E$, are estimated from smaller runs (Rosenfeld et al. 2020; Hoffmann et al. 2022). A larger exponent means that the corresponding excess-loss term falls faster as its input grows.

The power law applies to excess loss above the fitted floor, not to total loss. For example, holding $D$ large and fixed gives $L(N,D)-E-B/D^\beta\propto N^{-\alpha}$. That relationship is a straight line on log-log axes. The two-variable surface is not one universal straight line: model size, data size, and compute each produce a different one-dimensional slice or frontier.

Scaling studies fit this surface on a ladder of affordable runs and then test whether it predicts larger held-out runs. Figure 5.1 illustrates the basic extrapolation. The figure is schematic; a real study needs uncertainty estimates and held-out runs of its own.

Turning a compute budget into model and data sizes

For a dense decoder-only Transformer in a regime dominated by parameter-dependent matrix multiplications, training compute is often approximated by

$$C_{\mathrm{train}}\approx \kappa ND, \qquad \kappa\approx 6.$$

$C_{\mathrm{train}}$ is the number of training floating-point operations (FLOPs), $N$ is the number of parameters active for each token, and $D$ is the number of processed training tokens. The factor near six accounts roughly for a forward pass and its backward pass. It is a planning approximation, not a bill: attention, output layers, sparse routing, rematerialization, communication, and hardware utilization can all move actual time and cost away from it (Kaplan et al. 2020; Hoffmann et al. 2022).

At a fixed compute budget $C_{\mathrm{train}}$, choosing $N$ also fixes $D=C_{\mathrm{train}}/(\kappa N)$. Substituting that constraint into the loss fit gives

$$\begin{aligned} \widehat L_C(N) &=E+\frac{A}{N^\alpha}\\ &\quad+B\left(\frac{\kappa N}{C_{\mathrm{train}}}\right)^\beta. \end{aligned}$$

Increasing $N$ lowers the capacity term but leaves fewer tokens for training; decreasing $N$ does the reverse. This creates the U-shaped tradeoff in Figure 5.2. Under this particular loss and compute model, the minimum has a closed form:

$$\begin{aligned} N_\star= &\left(\frac{\alpha A}{\beta B}\right)^{\!1/(\alpha+\beta)}\\ &\quad\times \left(\frac{C_{\mathrm{train}}}{\kappa}\right)^{\!\beta/(\alpha+\beta)}, \end{aligned}$$

$$\begin{aligned} D_\star= &\left(\frac{\beta B}{\alpha A}\right)^{\!1/(\alpha+\beta)}\\ &\quad\times \left(\frac{C_{\mathrm{train}}}{\kappa}\right)^{\!\alpha/(\alpha+\beta)}. \end{aligned}$$

Here the star marks the loss-minimizing allocation predicted by the fit. These are the compute-optimal $N$ and $D$: the pair with the lowest predicted loss under the stated training-compute budget. The exponents determine how additional compute is divided. A constant token-to-parameter ratio emerges only when $\alpha=\beta$; it is not a universal consequence of power-law scaling.

Kaplan and Chinchilla estimated different frontiers

Kaplan et al. found smooth language-model scaling and estimated that the compute-optimal parameter count should grow approximately as $N_\star\propto C_{\mathrm{train}}^{0.73}$, leaving $D_\star\propto C_{\mathrm{train}}^{0.27}$ for data (Kaplan et al. 2020). Here $C_{\mathrm{train}}$ denotes the training-compute budget. That estimate favored increasingly large models trained on comparatively few tokens.

Hoffmann et al. ran a broader set of iso-compute experiments and found exponents near one half for both model size and data (Hoffmann et al. 2022). Their 70B parameter Chinchilla model was trained on 1.4T tokens, about twenty tokens per parameter, and outperformed several larger models trained with similar compute. The useful conclusion is that the models of that period were undertrained on data. “Twenty tokens per parameter” is the configuration of that demonstration and a practical rule of thumb, not a physical constant.

Later analyses help explain the disagreement without making it disappear. Kaplan counted non-embedding parameters, whereas Chinchilla used total parameters. Pearce and Song show that this accounting choice, combined with the smaller scale of Kaplan's experiments, reproduces much of the difference in the reported exponents (Pearce and Song 2024). Other experimental and fitting choices still matter.

The Chinchilla paper also used three estimation methods. Besiroglu et al. found that the published coefficients for its parametric third method fit reconstructed data poorly and conflict with the other two methods; the reported confidence intervals would require more than 600,000 runs, rather than the likely fewer than 500 (Besiroglu et al. 2024). Their refit again gives a near-even allocation, but with meaningful uncertainty. A scaling prescription should therefore name the fit, its experimental range, and its uncertainty instead of presenting one ratio as settled.

Deployment changes the objective

Chinchilla minimizes training loss for a fixed training-compute budget. A model that will serve many requests faces a different optimization problem because training is paid once while inference is paid repeatedly. A simplified lifetime cost is

$$\begin{aligned} C_{\mathrm{life}}(N,D,Q) &=C_{\mathrm{train}}(N,D)\\ &\quad+Q\,c_{\mathrm{serve}}(N). \end{aligned}$$

Here $Q$ is the expected lifetime volume of served tokens, or another consistently defined request unit, and $c_{\mathrm{serve}}(N)$ is the average compute or monetary cost per unit for a model of size $N$. For dense models, that cost generally grows with the active parameter count, although batching, sequence length, and hardware also matter.

Under a target-quality constraint and large expected demand, it can be cheaper overall to train a smaller model for longer than the training-compute optimum. Sardana et al. tested this reasoning across 47 models and found such allocations for scenarios with roughly a billion expected requests (Sardana et al. 2024). This is not a third way to divide the same fixed training budget. It changes the objective from lowest loss for fixed training FLOPs to lowest lifetime cost for fixed quality and demand.

Finite data changes the choice again

Inference-aware training assumes that more useful training tokens are available. Sometimes the binding constraint is the supply of unique high-quality text. In that case, a run must decide how often to repeat the corpus as well as how large to make the model.

Muennighoff et al. found little loss penalty from repeating constrained data for up to roughly four epochs at fixed compute in their experiments. Beyond that range, repeated tokens had diminishing value. Their fitted allocation still favored smaller models trained for more epochs than a naive application of the single-pass Chinchilla law would suggest (Muennighoff et al. 2023). The result is not that every fourth pass is free; it is evidence from a particular range of models, corpora, and compute budgets.

A 2026 preprint studies heavier repetition and adds an explicit overfitting penalty to the loss law (Lovelace et al. 2026). In experiments up to 1B parameters and 16 epochs, Lovelace et al. found regimes where further repetition became counterproductive and additional compute was better spent on capacity. Strong weight decay reduced their fitted repetition penalty by about 70%, but it worsened the single-epoch frontier and helped only after a data- and compute-dependent crossover. This is promising recent evidence, not yet a universal recipe.

The three planning questions are therefore distinct:

1. With fixed training FLOPs and enough fresh data, choose $N$ and $D$ to minimize held-out loss.
2. With a lifetime deployment objective, include expected serving demand and cost.
3. With fixed unique data, model the declining value and overfitting risk of repeated epochs.

How to run a scaling study

A useful fit starts with controlled experiments, not a formula copied from a different model family.

1. Fix the comparison. Keep the tokenizer, held-out distribution, data mixture, architecture family, optimizer, and loss definition consistent.
2. Build a scale ladder. Train several model sizes at several compute budgets, including multiple parameter-token allocations at each budget.
3. Use completed, comparable runs. A short run with the wrong learning-rate schedule can make a model look capacity-limited when it is merely under-optimized.
4. Hold runs out from fitting. A curve that explains its own input points but misses larger held-out runs is not a reliable extrapolator.
5. Fit more than one plausible form. Report sensitivity to exclusions, rounding, parameter definitions, and the assumed compute model.
6. Carry uncertainty into the allocation. Give a range of plausible $N$ and $D$, then test nearby choices instead of treating the numerical minimum as exact.

Maximal-update parametrization (muP) can reduce one part of the experimental cost. Under its width-dependent parameterization, many hyperparameter optima can transfer between different widths in a matched model family (Yang et al. 2021). It does not guarantee transfer across depth, architecture, data, optimizer, or batch changes; the target configuration still needs a validation run.

Test the allocation

The cell below evaluates the fixed-compute tradeoff with the rounded coefficients from the Besiroglu et al. refit. They are useful for reproducing this example, not as defaults for another training program. Change the exponents or compute budget and rerun the cell to see how the optimum moves.

import numpy as np
import matplotlib.pyplot as plt

# Rounded coefficients from Besiroglu et al. (2024).
E, A, B = 1.8172, 482.01, 2085.43
alpha, beta = 0.3478, 0.3658
C = 1e19
kappa = 6.0

def predicted_loss(N, D):
    return E + A / N**alpha + B / D**beta

N = np.logspace(7.5, 11, 400)
D = C / (kappa * N)
L = predicted_loss(N, D)
best_index = np.argmin(L)
best_N = N[best_index]
best_D = D[best_index]

plt.figure(figsize=(5, 3))
plt.plot(N, L, label="predicted held-out loss")
plt.axvline(best_N, color="tab:orange", linestyle="--", label="predicted optimum")
plt.xscale("log")
plt.xlabel("parameters N")
plt.ylabel("loss")
plt.legend()
plt.tight_layout()
plt.show()

print(f"N* = {best_N:.2e} parameters")
print(f"D* = {best_D:.2e} training tokens")
print(f"D*/N* = {best_D / best_N:.1f} tokens per parameter")

What the scaling fit does not tune

The loss surface chooses a model-data allocation only after the training recipe has been fixed. The optimizer, learning-rate schedule, and batch size need their own experiments.

Optimizer. AdamW decouples weight decay from Adam's gradient update and remains a common baseline (Loshchilov and Hutter 2019). It stores first- and second-moment estimates for each optimized parameter, so optimizer state is a major memory cost in large runs. Muon instead orthogonalizes momentum updates for matrix-shaped hidden-layer weights; it is not a curvature or second-order method. Its authors report about twice the compute efficiency of AdamW in their scaling experiments and use AdamW for embeddings, normalization parameters, and the output head (Liu et al. 2025). Kimi K2 later used a Muon variant in a model with roughly one trillion total parameters (Kimi Team 2025). These are strong demonstrations, not proof that one optimizer wins for every architecture or data mixture.

Learning-rate schedule. A cosine schedule assumes a planned end point and decays toward it. warmup-stable-decay (WSD) uses a warmup, a long stable plateau, and a decay that begins when training is about to stop. MiniCPM used this design to continue training from the plateau and compare several token budgets without restarting every run (Hu et al. 2024). The choice affects the validity of the scaling comparison: runs at different budgets must be evaluated at comparable points in their schedules.

Batch size. A batch is the set of token examples used for one optimizer update. Larger batches expose more data parallelism and can reduce the number of updates needed to reach a target loss. The benefit is not proportional forever. The gradient noise scale predicts a critical batch size near the knee where larger batches begin trading compute efficiency for fewer sequential steps (McCandlish et al. 2018). The noise scale often grows as loss falls, so a useful batch size may change during the run. Figure 5.3 shows the shape; it is not a hard failure threshold.

Keeping a long run stable

A predicted final loss is worthless if the run diverges before reaching it. Stability controls address different signals rather than forming one universal recovery recipe.

• z-loss penalizes the squared log of the softmax normalizer, discouraging all output logits from drifting together to large absolute values. PaLM used it as a stability control (Chowdhery et al. 2023).
• query-key normalization (QK-norm) normalizes query and key vectors before their dot product, limiting attention-logit growth. Small-scale proxy experiments reproduce this large-run instability and show that the mitigation can be tested cheaply (Wortsman et al. 2023).
• Global-norm gradient clipping rescales an unusually large gradient before it reaches the optimizer. Gradient norms, activation norms, and held-out loss still need to be monitored because clipping can hide a deteriorating run rather than repair its cause.
• Frequent checkpoints make hardware failures and diagnosed optimization failures recoverable. A restart policy must record the optimizer state, data position, and schedule position, not only the model weights.

Published reports illustrate why claims should be precise. DeepSeek-V3 reported no irrecoverable loss spikes and no rollbacks over its 14.8T-token run (DeepSeek-AI 2024). Kimi K2 reported zero loss spikes after adding QK-clip to Muon (Kimi Team 2025). Those are results from two training systems, not a guarantee that the same controls will transfer unchanged.

Distributed execution, checkpoint storage, numerical precision, and failure recovery are developed in Chapter 10. This chapter's boundary is the training recipe: the scaling fit proposes a destination, while optimization and systems engineering determine whether the run reaches it.

Before committing the run

The decision record for a large training run should state:

• the held-out distribution, tokenizer, and exact loss definition;
• the model family, parameter count convention, and training-token count;
• the measured scale ladder and which runs were held out from fitting;
• the fitted form, coefficient uncertainty, and extrapolation distance;
• whether the objective covers training FLOPs, wall-clock time, monetary cost, lifetime serving, or some combination;
• the unique-data budget and expected number of repeated epochs;
• the optimizer, schedule, batch policy, and stability gates.

That record turns a scaling law from an impressive curve into an auditable allocation decision. It also makes the forecast falsifiable: after the run, the observed held-out loss can be compared with the interval that justified the investment.

Further reading