Part IX · Chapter 70

Where Learning Hits Limits

AuthorChangkun Ou

Reading time~11 min

The previous chapters reached the limits of the physical machine. The next limit sits inside learning itself. The cheap resource that powered the last seven years of progress is nearly spent; every proposed substitute remains contested; a deployed model cannot learn from what it sees; and hallucination looks less like a bug to be patched than a property of the objective. The data wall, synthetic data, reinforcement learning on reasoning, test-time compute, continual learning, and hallucination are not six separate problems but one scientific question seen from six sides: what does a system learn, and stop learning, when it is trained to imitate a finite corpus and then frozen, and which of the field's escape routes change that answer rather than defer it.

Figure 70.1. Schematic of pressure on high-quality training data. Demand can keep rising after the easy stock is exhausted, pushing learning toward filtering, experience, and synthetic signal. Idealized curves, not measured data.

One number beneath the recipe

For most of the deep-learning era the recipe was simple: more compute, more parameters, more text, lower loss, as Chapter 5 laid out. That recipe depended on a resource that looked effectively free, the accumulated public text of the internet. The frontier problem is that the resource is finite. Epoch AI's estimate of the effective stock of quality-adjusted public human text is on the order of 300 trillion tokens, with wide uncertainty, and they project an 80 percent chance it is fully consumed somewhere between 2026 and 2032, pulled toward the near end of that window by the aggressive overtraining frontier labs already practice (Villalobos et al. 2024). Training compute keeps growing several- fold per year; the supply of high-quality human text is effectively flat. The two curves crossing is the data wall.

Everything else in this chapter follows from that crossing. Once you cannot simply add more human text, the only ways forward are to manufacture a substitute, learn from a signal other than text, spend more compute at inference instead of training (test-time compute), or extract more signal from the data you have. Each is a live research program, and each is contested precisely because it is trying to replace the missing source of cheap progress.

Figure 70.2. The data wall as the constraint under the chapter. Compute grows several-fold per year; the stock of high-quality human text is effectively flat. Where the curves cross, every escape route is an attempt to manufacture or substitute for the missing fuel, and each is contested.

Constraint arrow

The finite human-text stock dictates training strategy two layers up. It is why frontier runs overtrain far past the Chinchilla token-to-parameter ratio of Chapter 5 (roughly twenty training tokens per parameter at compute-optimal), why synthetic-data pipelines became central (Chapter 23), and why the marginal research dollar moved toward reinforcement learning and test-time compute as growth axes. A number about data supply reshaped the whole training recipe.

Can synthetic data replace human data?

The most direct substitute is synthetic data: text generated by models to train models. The hazard is model collapse. Training recursively on a model's own output degrades it, and a prominent result showed the degradation compounding across generations until the distribution narrows to noise (Shumailov et al. 2024). Read carelessly, that result says synthetic data is poison. Read carefully, it says something narrower and more useful: collapse happens when synthetic data replaces real data. When synthetic data accumulates alongside the real stock rather than replacing it, the error stays bounded (Gerstgrasser et al. 2024). The 2025 consensus is therefore conditional: synthetic data extends pretraining if it adds to human data rather than substituting for it, and production pipelines at trillion-token scale rely on exactly that distinction.

Two caveats keep this from being a settled answer. The first is knowledge collapse: even when the loss stays bounded, the synthetic regime can quietly lose facts while keeping fluency, so the model still sounds right while knowing less. The second cuts the other way and complicates the whole "fuel is spent" framing: a study of pretraining under effectively unlimited compute found that with heavy regularization and ensembling, a fixed data stock keeps yielding improvement, implying the wall is partly a question of data efficiency, not only data quantity (Kim et al. 2025). The wall may be further away for a lab willing to spend compute extracting more signal from the text it already has.

Learning from a signal instead of a corpus

The other route is to stop imitating text and start optimizing a reward. Reinforcement learning with verifiable rewards, the engine behind the reasoning models of Chapter 28, learns from signals that can be checked mechanically, a math answer that is correct, a unit test that passes, rather than from human-written tokens. Because the reward comes from the world rather than from a corpus, it is not bounded by the data wall in the same way.

The open scientific question is sharper than the engineering success suggests: does this reinforcement learning create new reasoning ability or only elicit ability the base model (the raw pretrained model before any post-training) already had? One body of evidence points to elicitation. A reinforcement-trained model beats its base model when each is allowed a single attempt, but when both are allowed hundreds of attempts the base model catches up and passes it, which suggests the training sharpens the sampling of solutions the base model could already find rather than adding solutions it could not (Yue et al. 2025). The opposing evidence points to creation: prolonged, stabilized reinforcement learning has been reported to lift performance even at many attempts, solving problems the base model fails at any budget (Liu et al. 2025). The most current reading is that the metric at the center of the dispute is itself suspect, since the many-attempts measure may capture how widely a model explores rather than the boundary of what it can do, which means the elicitation-versus-creation question is not settled but reopened. Test-time compute, spending more inference on longer chains or search, is the third axis, and it too has a contested ceiling: returns diminish, and past a point longer reasoning can abandon a correct answer, so more thinking is not monotonically better.

The model that cannot learn from what it sees

A deployed frontier model is frozen. Its weights do not change in response to anything it encounters after training; all post-deployment adaptation lives in the prompt, the context window, or an external store, never in the parameters. This is the starkest gap between these systems and any everyday notion of learning, and it is not a choice so much as a constraint. Updating the weights of a deployed model on new information tends to overwrite old information, because knowledge is entangled across billions of parameters and a fine-tune (continued training on a narrower dataset) reshapes the whole landscape. This is catastrophic forgetting, and it remains unsolved at the frontier.

The research directions are real but early. Sparse, memory-targeted fine-tuning updates a small slice of parameters to add knowledge with less disruption to the rest (Lin and others 2025), and architectural proposals treat the model as nested levels of memory updated at different rates, reporting much higher retention of old tasks than ordinary fine-tuning. None of this yet adds up to a frontier model that learns continuously from its own deployment.

Constraint arrow

Because the weights cannot update online, capability persistence is pushed up the stack. Everything a model needs to "remember" after training must live in context, in retrieval, or in an external memory system, which is why Chapter 39, Chapter 44, and Chapter 46 exist as substantial engineering layers. The frozen-weights limit at the learning layer manufactures the orchestration layer above it.

Hallucination as a property, not a bug

The residual error of these systems has a name and, increasingly, a theory. A 2025 argument reframed hallucination from a defect to be patched into a statistical consequence of the training objective: a model trained to estimate the density of a corpus and then to generate from it will, on inputs where the truth is underdetermined by what it learned, produce confident falsehoods, because the objective never rewarded saying "I do not know" (Kalai et al. 2025). The argument has a second, sharper half: hallucination persists because the benchmarks that grade models reward a confident guess over an honest abstention, so the evaluation layer trains the very behavior it then measures. Later information-theoretic work tied a floor on hallucination to the model's compression of its training data, strengthening the case that some residual is structural rather than fixable.

The opposing camp is not naive. Calibration-aware training, abstention rewards, and retrieval grounding measurably reduce hallucination on narrow tasks, sometimes sharply, and the practical rate is far from an immovable constant. The disagreement is about the limit: whether the residual can be driven to zero, or whether the generative objective guarantees a floor that mitigation can only relocate.

What's contested

Hard data wall or movable? One camp holds high-quality human text is effectively flat and the cheapest gains are gone. Another counters that accumulated synthetic data plus data-efficiency gains move the wall indefinitely. The collapse-versus-accumulation result sits between them: replacement diverges, accumulation stays bounded, and knowledge collapse may still creep in.
Does RL create reasoning or elicit it? The elicitation camp reads the many-attempts crossover as proof that reinforcement learning only sharpens what the base model can already sample. The creation camp reads prolonged training as expanding the boundary. The current twist is that the deciding metric is itself contested, so the question is open rather than answered.
Does pretraining still pay? A plateau camp sees raw scaling returns flattening. A still-pays camp notes that overtraining a smaller model past Chinchilla pays because inference is paid forever. An axis-shift camp says the growth moved to reinforcement learning and test-time compute, and pretraining is now a substrate rather than the engine.
Is hallucination reducible to zero? The structural camp says the generative objective guarantees a floor. The fixable camp answers that calibration and grounding keep lowering the rate. The disagreement is about the limit, not the direction.
Can a frozen model be intelligent? One camp says context and retrieval substitute for weight updates and that freezing is a safety feature. Another says real intelligence requires lifelong adaptation and that catastrophic forgetting is the core unsolved obstacle.

Learning after the cheap corpus

At this layer, capability is no longer a simple function of scale. It is the ability to learn more from less, from signals rather than corpora, and from compute spent at inference. Efficiency becomes a scientific variable rather than a deployment concern: how many useful behaviors a fixed data stock can yield now decides how far the wall recedes. Trust is bounded by the hallucination floor and frozen weights, the two places where the system's honesty and adaptability are limited by the objective and architecture rather than by effort. The next chapter carries the same question out of text, into pixels, video, and the physical world, where the data wall reappears in a harsher form: there was never an internet of the data these systems need.