Scaling Laws and Compute Allocation

The first irreversible choice in the stack is the training run itself. A model's architecture and its mixture of training data are committed to a cluster, and weeks of computation turn into a finished base model. You cannot run the whole thing twice to see which choice was better, so the entire problem is to decide, before booking the cluster, how large to make the model and how much text to train it on. This chapter is about how that decision is made and then carried out.

Four quantities run through everything below, so it helps to fix them in plain terms first. A parameter is one of the adjustable numbers inside the model, set during training; model size means how many of them there are, usually billions. A token is a small chunk of text, roughly a short word or a word-piece, and a model learns by reading a long stream of them. Loss is a single number that measures how badly the model predicts the next token, averaged over all the text it reads: lower is better, and pushing it down is the whole point of the run. Compute is the total arithmetic the run costs, measured in FLOPs (floating-point operations); for a transformer it is close to six times the parameter count times the number of training tokens. With those in hand the chapter's question is simple to state: spend a fixed compute budget so the final loss is as low as it can be, and know roughly what that loss will be before committing.

The irreversible bet

A pre-training run commits a cluster for weeks and a budget you cannot refund. Before it starts you must choose how many parameters the model will have, how many tokens it will read, and the settings that govern the optimization (the learning-rate schedule and the batch size, both defined later in this chapter). You must choose them from small experiments, because you cannot afford to try them once at full scale. Everything that follows is a way of answering that same question from cheap runs, before the cluster is booked and the bet is locked in.

The question is answerable because the loss has a predictable shape, and it has that shape because of how the model is trained. The model is trained to predict the next token in a stream of text, one token at a time, each prediction conditioned on the tokens before it; this is next-token prediction. Every position in the text is therefore its own training example, with the right answer being simply the token that actually came next, so the text supervises the model for free and at enormous scale. The loss is the average surprise of those predictions: formally it is cross-entropy, the number of bits, or nats if measured in natural-log units, the model spends to encode each true next token, and perplexity is just its exponential. That single scalar, the loss, is what the run buys, and the rest of the chapter learns to forecast it and then to minimize it per dollar.

A law you can extrapolate

The decision that dominates everything else is sizing: how to split the budget between parameters and tokens. It is tractable because the loss falls in a regular way as either one grows, a power law. A power law is a relationship where the loss drops by a constant factor each time the input grows by a constant factor, which is exactly the relationship that plots as a straight line when both axes are logarithmic. A scaling law fits that line on a ladder of small, affordable runs and reads off where it goes for a run you can only afford once (Kaplan et al. 2020). The form that is actually fit pairs two power-law terms with an irreducible floor $E$, the loss that no amount of scale removes (Rosenfeld et al. 2020; Hoffmann et al. 2022):

$$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$$

Here $N$ is the parameter count and $D$ is the number of training tokens. The two exponents $\alpha$ and $\beta$ set how fast loss falls as the model and the data grow, a larger exponent meaning a steeper return to scale, and $A$ and $B$ are scaling constants; all four are fit from the cheap runs. With them in hand you read off the compute-optimal $N$ and $D$: the pair with the lowest predicted loss for the budget you intend to spend. One subtlety: training tokens are token instances read during training, not distinct vocabulary entries, so a second pass over the same text counts again.

The law matters because it extrapolates. A few cheap runs stand in for a run you will only ever do once: because the loss is a power law, those runs fall on a straight line in log-log coordinates, and the line predicts the loss far to its right, as Figure 5.1 sketches. Compute here is the total spend, close to $6ND$, so the same straight line can be read against parameters, against tokens, or against the whole budget.

The same logic extends to the model's other tuning knobs, the hyperparameters that the loss formula does not name: settings like the learning rate (how big a step each update takes) and the initialization (the random values the parameters start from). These normally have to be re-tuned at every model size. Maximal-update parametrization (muP) removes that cost: it parametrizes the network's width (how many parameters sit side by side in each layer) so the best learning rate and initialization stay fixed as the model grows, which lets you tune them once on a small proxy and carry them to full scale (Yang et al. 2022). Sizing and tuning both become decisions you make on small runs and trust at large ones.

Why a power law at all

This section is optional theory. It explains why loss falls so smoothly, and can be skipped without losing the chapter's practical thread. The law is an empirical regularity, not a theorem, and asking where it comes from bounds how far to trust it. Two derivations reach it from different starts. One reads the slope off the data. Picture the data as points lying on a curved surface, a manifold, whose effective number of independent directions is its intrinsic dimension $d$; a model fitting that surface has loss that scales as about $N^{-4/d}$, so the more independent directions the data has, the slower scale pays off (Sharma and Kaplan 2020). The other reads it off the skills: if competence decomposes into discrete units learned in order of decreasing frequency, and those frequencies are themselves power-law distributed, then learning them in order produces a smooth power-law loss out of strictly discrete steps (Michaud et al. 2023). A percolation argument on natural data recovers both as two regimes of one model (Brill 2024). None of this is settled, but the shared implication is concrete: the slope is a property of the data, not the architecture.

That discrete-skills reading connects to a debate the book returns to later. If skills are acquired in discrete steps, the overall loss curve can stay smooth while individual abilities switch on suddenly, and whether scaling then looks gradual or shows a sharp "emergent" jump turns largely on how you measure it. Chapter 54 makes the discreteness concrete, as separable features recoverable from a trained model, and Chapter 3 takes up the emergence question.

Kaplan, then Chinchilla

The sizing rule itself has a history, and getting it wrong is the most expensive mistake in this chapter. Kaplan et al. (2020) established that loss follows smooth power laws and, from their fits, recommended spending most of a larger budget on parameters (Kaplan et al. 2020). Hoffmann et al. (2022), the Chinchilla paper, corrected this: parameters and tokens should scale together, roughly twenty training tokens per parameter at compute-optimal, and the Kaplan-era recipe had been systematically under-training on data (Hoffmann et al. 2022). A Chinchilla-optimal model is smaller and reads more than the prior generation expected.

That two careful teams disagreed at all is instructive, and the gap turned out to be mostly an accounting choice. Some of a model's parameters sit in its embeddings, the lookup table that maps each token to a vector, one row per vocabulary entry; the rest are the layers that do the actual computation. Kaplan counted only the non-embedding parameters, and at small scale the embeddings are a large enough share of the total to tilt the optimal split toward parameters. Count total parameters instead and let scale grow, and the exponent converges to Chinchilla's roughly equal split (Pearce and Song 2024). The disagreement was never nature being ambiguous, but which quantity the curve was plotted against.

The same compute budget, fixed by $C \approx 6ND$, splits very differently under each of these recipes. Figure 5.2 contrasts the three allocations of one budget: Kaplan spends it mostly on parameters, Chinchilla balances parameters against tokens at roughly twenty tokens per parameter, and inference-aware over-training deliberately shrinks the model and pours the rest into tokens so each served token is cheaper.

The recipe did not stop at Chinchilla. Chinchilla assumes you can always feed the model more fresh text, but the supply of high-quality unique text is finite, and a large run can exhaust it. Once that happens, data-constrained scaling takes over: re-reading the same text has decaying value, and past a few epochs (a few full passes over the data) adding parameters beats adding repetitions (Muennighoff et al. 2023). Later work sharpens the penalty rather than the schedule, adding an explicit overfitting term, where overfitting means the model starts memorizing the training text instead of learning patterns that generalize. That term grows with the ratio of model capacity to unique data, so larger models exhaust repeated tokens faster, with strong weight decay (a steady pull that keeps the parameters small and so discourages memorization) buying part of it back (Lovelace et al. 2026). The power-law lens also travels beyond pretraining loss. The same forecast-from-cheap-runs logic now sizes test-time compute (Chapter 30), reduced precision (Chapter 34), and distillation (Chapter 29), each its own curve under its own budget, so reading scaling as a single pretraining decision understates how often the field makes this bet. The frontier of the recipe is still moving.

Test it yourself

The law is concrete enough to run. The cell below walks a fixed compute budget along the parameters-versus-tokens trade and finds where the loss bottoms out. Edit the exponents and rerun (Run, or Ctrl/Cmd+Enter) to watch the compute-optimal split move.

import numpy as np, matplotlib.pyplot as plt

# Toy scaling law L(N, D) = E + A/N^a + B/D^b. Change the exponents.
E, A, B, a, b = 1.7, 400.0, 410.0, 0.34, 0.28
loss = lambda N, D: E + A / N**a + B / D**b

C = 1e19                       # fixed compute budget, roughly 6 * N * D
N = np.logspace(8, 11, 300)    # candidate parameter counts
D = C / (6 * N)                # tokens the budget then allows

plt.figure(figsize=(5, 3))
plt.plot(N, loss(N, D))
plt.xscale("log"); plt.xlabel("parameters N"); plt.ylabel("loss")
plt.tight_layout(); plt.show()

best = N[np.argmin(loss(N, D))]
print(f"compute-optimal: N ~ {best:.2e} params, D ~ {C/(6*best):.2e} tokens")

How much to trust the fit

The toy makes plain something the published curves tend to hide: the extrapolation is only as trustworthy as the fit, and the fit is delicate. Re-deriving Chinchilla's parametric estimate from the same data, a later replication found it inconsistent with the paper's own other two methods, with confidence intervals so tight they would imply hundreds of thousands of runs behind a few hundred (Besiroglu et al. 2024). It also fits the data points reconstructed from the original plots poorly. The lesson is fragility: ordinary choices in how a curve is fit can move where it extrapolates to, and none of them leave a mark on the final power law you read off. This does not make the law useless, it remains the best forecast available and is bet on routinely, but it does make a scaling law a measurement instrument, with error bars the headline number omits. The same caution returns wherever the field reads a curve to settle a costly question, from benchmark scores in Chapter 48 to capability forecasts in Chapter 72.

The rest of the recipe

Once the sizing rule fixes $N$ and $D$, the problem shifts from forecasting the run to executing it. The scaling law does not say how to actually walk the parameters down to that loss, and those choices are a set of balances, each with a knee.

The optimizer is the first of them: the algorithm that turns each batch's gradient, the direction that would lower the loss, into a concrete update to the parameters. AdamW is the default. It keeps two running averages per parameter (the moment estimates, tracking the recent mean and spread of the gradient), and that extra per-parameter state is the dominant memory term, which is why Chapter 10 shards it across devices (Loshchilov and Hutter 2019). Second-order and preconditioned optimizers, Shampoo and the newer Muon, use curvature information to condition the update, aiming at better loss per FLOP (Jordan et al. 2024). Muon has since trained Moonlight, a 16B MoE model, over 5.7T tokens (Liu et al. 2025), and then Kimi K2 at roughly a trillion total parameters (Kimi Team 2025), which makes it a frontier-proven option even though AdamW remains the default. The learning-rate schedule that drives the optimizer has three parts: a linear warmup while the moment estimates are still cold, a stable high phase, and a decay. Cosine decay to a small floor is the well-understood default, but it locks the token budget at run start. warmup-stable-decay (WSD) holds a constant rate on a plateau and decays only when you choose to stop, which makes one plateau checkpoint spawn many decays and makes scaling experiments composable (Hu et al. 2024). It trades a small loss penalty for the ability to keep training, fork decays, and run clean scaling ladders. Choose cosine for a single known-budget run, the plateau schedule when the budget is uncertain.

Batch size is the second balance. A larger batch buys data parallelism and shorter wall-clock, but past the critical batch size you burn compute for no loss gain, and too small a batch leaves the accelerators idle and pays communication overhead per useful token (McCandlish et al. 2018). Figure 5.4 sketches that knee: progress per step climbs almost linearly while the batch is small, then flattens once it passes the critical size. The critical batch size grows during training, so the optimal batch is a schedule, not a constant.

And muP closes the loop back to sizing. Tuning the learning rate and initialization on a small proxy and transferring them at full width is what keeps the largest run from leaving loss on the table for a reason you would never see otherwise (Yang et al. 2022).

Keeping the run alive

The plan only matters if the run stays alive. Multi-week training therefore needs a small operational kit alongside the scaling rule. A few of these techniques reach into the internals of the transformer, covered in Chapter 8; the point here is only that each one guards against a specific way a long run can blow up. z-loss adds a small penalty that keeps the model's raw output scores, the logits that get turned into next-token probabilities, from drifting ever larger (Chowdhery et al. 2022). query-key normalization (QK-norm) normalizes two of the internal vectors in the attention mechanism, the query and the key, before they are multiplied together, to stop their product from growing without bound. Depth-scaled initialization sets the starting parameter values with the model's depth in mind, and global-norm gradient clipping caps the size of any single update as a blunt safety net. Loss-spike recovery is a procedure, not a hyperparameter: detect the spike from gradient-norm and loss telemetry, then skip and resume from a recent checkpoint, lower the rate, and reorder the data batches that triggered it (DeepSeek-AI 2024). Figure 5.5 traces that loop.

The failure modes are worth naming because each bites at a different time. Divergence to NaN sends the loss to infinity early, usually from too high a rate, too short a warmup, or a precision interaction from Chapter 10, and is cheapest to prevent with warmup and clipping. Loss spikes appear mid-run and waste days of compute if you have no recovery plan. The wrong size-versus-tokens choice is the most expensive of all because it is the whole run, and an untuned learning rate at full width loses gains you cannot detect without a muP baseline to expose them.

Further reading