Where Learning Hits Limits

The previous part ended with a machine that must keep working while its parts fail. Learning has a similar requirement: its limits must be stated at the right boundary. A finite public corpus is a resource limit. Next-token prediction imposes an objective limit. A released checkpoint creates an adaptation boundary. A system that cannot support or safely abstain from an answer reaches an evidence limit. These limits interact, but they are not interchangeable.

The cheap resource that powered much of modern language-model progress was public human text. The discussion still covers synthetic data, reinforcement learning on reasoning, test-time compute, continual learning, and hallucination, but it does not treat them as one problem. A model trained to imitate a finite corpus can later receive synthetic examples, checked experience, retrieved evidence, or parameter updates. Each route changes a different boundary. Can learning continue past the public-text boundary? The useful question is which routes change that answer rather than defer it, and which boundary each route moves.

The reliability chapter used a failure contract. A learning limit needs the same discipline: name the task distribution, information source, update rule, compute budget, and acceptance test before calling a result a wall.

A stock forecast is not a fuel gauge

Villalobos and colleagues made a conditional forecast about public human text, not a measurement of a tank becoming empty. Their 2024 ICML paper estimated the raw stock of the indexed web at a median 510 trillion tokens, with a very wide 95 percent interval from 130 trillion to 2.1 quadrillion. They then adjusted for filtering quality and the limited value of repeating data. The resulting effective stock at the modeled crossing was roughly 400 trillion tokens (Villalobos et al. 2024).

The paper projected historical growth in notable training dataset size at about 2.4 times per year. Under its mixture of historical and compute-constrained projections, one run's effective dataset demand reaches the modeled stock from 2026 to 2032, with a median crossing year of 2028. A hypothetical fivefold overtraining policy moves the crossing about one year earlier; the conclusion allows one or two years earlier. This is a forecast, not an observation. The paper assumes continued growth, uncertain web expansion, a particular quality adjustment, and a model of repeated epochs. It says the stock becomes fully utilized by one run, not physically consumed.

Three token counts must remain separate:

• unique retained tokens count the deduplicated examples admitted to a dataset;
• dataset size counts that retained corpus once; and
• training-token exposures count every token instance presented during training, including repetitions across epochs.

The following small model exposes why a crossing year is sensitive to its inputs:

$$D(t)=D_0 g^{t-t_0}, \qquad t^* = t_0+\frac{\ln(S/D_0)}{\ln g}.$$

where:

• $D(t)$ is projected dataset demand in year $t$, measured in tokens;
• $D_0$ is dataset demand in the reference year $t_0$;
• $g>1$ is the assumed annual growth factor;
• $S$ is the assumed usable stock in the same token units;
• $t^*$ is the year at which the simple model gives $D(t^*)=S$; and
• $\ln$ is the natural logarithm.

This equation assumes a fixed $S$ and constant exponential growth $g$. The published forecast uses a richer uncertainty model. This one is only a sensitivity calculation.

from math import log

stock_trillion = 400.0
start_trillion = 15.0
start_year = 2024.0

for annual_growth in (2.4, 1.8, 3.0):
    crossing = start_year + log(stock_trillion / start_trillion) / log(annual_growth)
    print(f"{annual_growth:.1f}x/year crossing: {crossing:.2f}")

The 15T starting point and 400T stock are illustrative inputs taken from the paper's reported 2024 scale and effective crossing stock. At 2.4x annual growth, this simple calculation gives late 2027. Slower growth moves the answer by years. It does not reproduce the paper's Monte Carlo forecast.

Synthetic data changes the protocol

Synthetic data can improve format, coverage, difficulty, or density. It does not automatically add information beyond its teacher, prompt source, tools, or verifier. The first question is therefore not “real or synthetic?” but “what recursive protocol generated the next training set?”

A replacement protocol discards the earlier data and trains generation $n+1$ on samples from generation $n$. An accumulation protocol retains the human anchor and earlier generations:

$$\mathcal{A}_n = \mathcal{H} \cup \bigcup_{i=0}^{n-1}\mathcal{G}_i.$$

where:

• $n$ is the index of the generation being trained;
• $\mathcal{H}$ is the retained human-data anchor;
• $\mathcal{G}_i$ is the synthetic dataset generated after fitting generation $i$;
• $\mathcal{A}_n$ is the accumulated dataset used to fit generation $n$;
• $i$ indexes a prior generation; and
• $\cup$ and $\bigcup$ denote ordinary and repeated set union. In a real pipeline these are usually multisets, so mixture weights and deduplication must also be recorded.

Shumailov et al. studied recursive model-data feedback and showed why low probability tail events disappear first. Their language-model evidence was not a frontier pretraining run: it repeatedly fine-tuned OPT-125M on WikiText-2 continuations generated with five-way beam search. The tested replacement setting degraded perplexity; retaining 10 percent of the original data reduced the degradation. Late convergence toward a point distribution came from the paper's analytical, Gaussian-mixture, and VAE settings, not from a frontier language model turning into noise (Shumailov et al. 2024).

Gerstgrasser et al. compared replacement and accumulation in small GPT-2 and Llama 2 experiments on TinyStories. Accumulation avoided divergent degradation over the tested generations. Their finite upper bound is a linear-model theorem under assumptions that are much narrower than modern pretraining (Gerstgrasser et al. 2024). This does not prove that retaining old data prevents every kind of collapse, bias amplification, contamination, or loss of rare knowledge at frontier scale.

Positive synthetic-data results need equally careful scope. BeyondWeb used targeted rephrasing in mixtures containing 60 percent RedPajama and 40 percent synthetic text. A 1B model trained for one trillion tokens; 3B and 8B models trained for 180 billion tokens. Across 14 benchmarks, the paper reports up to 5.1 percentage points over Cosmopedia and up to 7.7 times faster training than its open-web baseline in a particular matching comparison (Maini and others 2025). Those results support targeted rephrasing in that pipeline. They do not establish that arbitrary self-generated text can replace human evidence.

Fixed data can also yield more value without inventing more documents. Kim et al. trained on seed stocks from 200M to 1.6B tokens and found that stronger regularization and ensembling improved the fitted loss asymptote. Their headline 5.17 times data-efficiency result at 200 million tokens is a nested scaling-law extrapolation; the best directly observed ensemble result was smaller. The curves approach finite asymptotes, so this is not unlimited improvement from a fixed corpus (Kim et al. 2025).

A synthetic-data record should therefore retain the source provenance, teacher checkpoint, generation policy, prompt source, temperature and filters, human-data anchor, deduplication policy, mixture weights, verifier version, and held-out human evaluation. Without that record, “synthetic” describes an origin, not a learning signal.

Checked experience changes the supervision bottleneck

reinforcement learning with verifiable rewards (RLVR), reinforcement learning with verifiable rewards, replaces a human-written solution trace with an automatic checker where one exists. A math equality, compiler, unit test, or formal proof checker can label many sampled attempts cheaply. This changes the supervision bottleneck described in Chapter 21 and Chapter 28. It does not remove the need for tasks, a correct checker, exploration compute, or a held-out evaluation. Incomplete reward coverage still permits reward hacking (reward hacking).

The elicitation-versus-creation debate turns heavily on pass@k. The following formula represents coverage for a fixed problem $x$, decoder, and verifier:

$$p = \Pr[v(x,Y)=1], \qquad \operatorname{pass}@k(x) = 1-(1-p)^k.$$

where:

• $x$ is one problem;
• $Y$ is one sample from a fixed decoder distribution for $x$;
• $v(x,Y)\in\{0,1\}$ is a fixed verifier that accepts or rejects the sample;
• $\Pr$ denotes probability;
• $p$ is the one-sample probability of verifier acceptance;
• $k$ is the number of independent samples; and
• $\operatorname{pass}@k(x)$ is the probability that at least one of those samples is accepted.

With $n\ge k$ sampled candidates and $c_x$ accepted candidates, Chen et al. estimate the quantity without replacement as

$$\widehat{\operatorname{pass}@k}(x) = 1- \frac{\binom{n-c_x}{k}}{\binom{n}{k}}.$$

Here $n$ is the sampled pool size, $c_x$ is its accepted count, $\binom{a}{b}$ is the number of $b$-element subsets of an $a$-element set, and the hat marks an estimate (Chen and others 2021). Both expressions assume exchangeable or independent samples from a fixed decoder and a trustworthy verifier. This is a coverage metric, not the reliability of one returned answer. A deployed system still needs a selection rule. High pass@k cannot identify which candidate is correct, and zero observed successes at finite $k$ does not prove that $p=0$.

Yue et al. evaluated six RLVR algorithms across math, coding, and visual reasoning in a NeurIPS 2025 study. In their tested setups, post-RL checkpoints won at low $k$, while the starting checkpoint before the RL stage caught up or passed them at larger $k$. The result supports an elicitation interpretation for those model families, prompts, decoders, and budgets (Yue et al. 2025). It is not a theorem that RL cannot create a new policy.

ProRL used a different training protocol: prolonged RL with KL control, reference resets, diverse tasks, and a 1.5B distilled reasoning starting checkpoint. It reports gains through a candidate budget of 256, including tasks with no observed starting-checkpoint success among 256 samples (Liu et al. 2025). That finite zero is not failure at every possible budget. The two studies expose a real empirical disagreement. The comparison does not settle an absolute capability boundary.

Test-time compute needs a way to choose

Test-time compute includes at least two different operations: generate more parallel candidates, or spend more sequential tokens revising one trajectory. Both require a generator, a verifier or judge, and a selector that returns one accepted result. More candidates help only if the generator covers a useful answer and the selection mechanism can recognize it.

Snell et al. compared best-of-N sampling and verifier-guided revision on MATH. The best method changed with problem difficulty and budget. Their compute-aware allocation was more than four times as efficient as the tested best-of-N baseline at reported points. In a FLOPs-matched comparison, a smaller model could outperform a 14 times larger model where the smaller model already had nontrivial success probability (Snell et al. 2025). The result depends on PaLM 2 model variants, specialized revision and verifier models, a math benchmark, and a particular accounting of training and inference FLOPs. It is not a general scaling law saying inference compute always substitutes for parameters.

Deployment has three kinds of mutable state

A served checkpoint commonly keeps parameters fixed during a request, but frozen weights are a release policy, not a theoretical inability to update a deployed model. The following formula represents a state model that separates parameters, session context, and external memory:

$$Y_t \sim p_{\theta_t}(\,\cdot\mid x_t,c_t,m_t), \qquad (c_{t+1},m_{t+1}) = V(c_t,m_t,x_t,Y_t), \qquad \theta_{t+1} = U(\theta_t,e_t).$$

where:

• $t$ indexes an interaction or update interval;
• $x_t$ is the current input;
• $Y_t$ is the sampled output;
• $p_{\theta_t}$ is the model distribution parameterized by weights $\theta_t$, and $\cdot$ denotes the possible output value;
• $c_t$ is temporary session or context state;
• $m_t$ is mutable external memory;
• $V$ is the versioned rule that updates context and external memory;
• $e_t$ is a reviewed batch of new training experience; and
• $U$ is the parameter-update procedure that produces the next checkpoint $\theta_{t+1}$.

In-context learning changes $c_t$. Retrieval or a memory service changes $m_t$. Continual pretraining or fine-tuning changes $\theta_t$. Calling all three “learning” hides different persistence, privacy, poisoning, evaluation, and rollback properties.

Sequential parameter training has long exposed the stability-plasticity problem: acquiring the new distribution can degrade performance on an old one. Kirkpatrick et al.'s 2017 elastic weight consolidation work protected parameters important to earlier tasks, establishing one influential mitigation rather than a universal solution (Kirkpatrick and others 2017). Operationally, measure both acquisition and retention:

$$A_{\mathrm{new}} = Q_{\mathrm{new}}(\theta_{\mathrm{after}}) - Q_{\mathrm{new}}(\theta_{\mathrm{before}}), \qquad \Delta_{\mathrm{old}} = Q_{\mathrm{old}}(\theta_{\mathrm{after}}) - Q_{\mathrm{old}}(\theta_{\mathrm{before}}).$$

Here $Q_{\mathrm{new}}$ and $Q_{\mathrm{old}}$ are fixed evaluation scores on new and old held-out distributions; $\theta_{\mathrm{before}}$ and $\theta_{\mathrm{after}}$ are the checkpoints before and after the update; $A_{\mathrm{new}}$ measures acquisition; and $\Delta_{\mathrm{old}}$ measures retention change. A large positive acquisition score does not compensate for an unreported negative retention score. Catastrophic interference is a large negative retention change under a declared old-task suite, not a diagnosis of why it occurred.

Lin et al. provide a scoped recent result. They replaced one feed-forward layer in a 1.3B model with a large sparse memory pool and evaluated two question-answering tasks. At the same level of new-knowledge acquisition, NaturalQuestions F1 fell 89 percent after full fine-tuning, 71 percent with LoRA, and 11 percent with sparse updates to the memory-layer models (Lin et al. 2025). This does not demonstrate frontier-scale open-ended continual learning, retention of reasoning and safety, or safe learning from unreviewed production traffic.

Live parameter updates also create a security boundary. In one red-team study, ten adversarial fine-tuning examples compromised safety behavior in the tested GPT-3.5 Turbo setup; benign fine-tuning also caused smaller regressions (Qi et al. 2024). A production update path therefore needs provenance, quarantine, replay or rehearsal data, a retention suite, safety and privacy tests, a promotion gate, checkpoint versioning, canarying, and rollback. Learning continuously does not require promoting continuously.

This boundary explains why Chapter 44 and the memory systems above it remain important: external state can be updated quickly while the model release stays testable. It is an engineering trade, not proof that parameters should never learn online.

Hallucination needs an evidence boundary

“Hallucination” is often used for several different failures: a plausible but false closed-book statement, an answer unsupported by declared retrieved evidence, a contradiction of the prompt, or an unverifiable guess. Those cases need different tests. Chapter 51 develops the evidence contract; here the important point is that the error rate depends on task distribution, available evidence, and the option to abstain.

Kalai and Vempala proved a lower bound for certain arbitrary facts under a generative-calibration model. The result does not apply in the same way to systematic facts or facts repeated in training (Kalai and Vempala 2024). Kalai and colleagues later separated two mechanisms: statistical errors in a closed-book base model, and evaluation incentives that reward guessing when a correct answer scores 1 while both a wrong answer and “I don't know” score 0 (Kalai et al. 2025). Binary scoring can shape post-training, model selection, and leaderboards. Merely running an evaluation does not itself update a model.

Most importantly, hallucination is not inevitable for every complete system. The authors explicitly allow systems that use retrieval, a database, a calculator, or abstention outside a bounded domain. It may return fewer answers, so error must be reported together with coverage (Geifman and El-Yaniv 2017). A zero-error system that abstains on everything is useless; a high-coverage system that emits confident error is unsafe.

The following expected-utility formula represents the acceptance decision:

$$\mathbb{E}[U(\mathrm{answer})\mid x] = q(x)u_{\mathrm{correct}} + [1-q(x)]u_{\mathrm{wrong}},$$

and answer only when

$$q(x) \ge \frac{u_{\mathrm{abstain}}-u_{\mathrm{wrong}}} {u_{\mathrm{correct}}-u_{\mathrm{wrong}}}.$$

where:

• $x$ is the current query together with its available evidence;
• $q(x)$ is a calibrated estimate that the proposed answer is correct, so $1-q(x)$ is the estimated probability that it is wrong;
• $U$ is decision utility and $\mathbb{E}$ denotes its expectation;
• $u_{\mathrm{correct}}$ is the utility of a correct answer;
• $u_{\mathrm{wrong}}$ is the utility of a wrong answer;
• $u_{\mathrm{abstain}}$ is the utility of abstaining; and
• the inequality assumes $u_{\mathrm{correct}}>u_{\mathrm{abstain}}> u_{\mathrm{wrong}}$.

An accuracy-only benchmark effectively gives abstention the same value as an error, lowering the acceptance threshold and encouraging guesses. A high-stakes system can assign a much worse utility to a confident error, raising the threshold. Calibration, evidence retrieval, and utility are all deployment-specific, so there is no single universal hallucination floor.

Make every escape route falsifiable

A learning-limit review should compare routes on one ledger:

Every row also needs a timestamp, a held-out distribution protected from optimization, and an owner for failed evidence. Reporting a better score without the changed information source, compute budget, or selection policy does not show which boundary moved.

The limit is an interface

A limit is useful only when its boundary is named. Public text can become scarce without learning ending. Synthetic data can improve efficiency without creating an independent source of truth. RL can change a policy without proving an absolute capability boundary. Test-time compute can increase candidate coverage without selecting the right candidate. Parameter updates can acquire facts while damaging retention. Abstention can eliminate emitted falsehoods by declining useful work.

The next chapter, Chapter 71, asks how far capability has moved. This chapter supplies its prerequisite: a claim about progress must say what information entered, what state changed, what compute was spent, and what test accepted the result.

Further reading