Borrowed Ideas: What AI Took From Other Sciences

The field rarely invents from nothing. Many of its central algorithms were borrowed from older sciences: information theory, physics, neuroscience, psychology, biology. The useful distinction is between two kinds of borrowing. Some borrowings became shared mathematics: the AI object and the original are provably the same equation. Others never got past shared vocabulary, a word taken without the mechanism behind it. Each of the stronger ones moves through the same three steps, from inspiration to formalization to the departure where engineering keeps what computes and drops the rest. That distinction will matter throughout the book, because some cross-disciplinary intuitions carry weight as mechanism and others are only pictures that help organize thought.

Compression is prediction

The deepest borrowing is also the most exact, and it is the one running under every model in this book. Shannon gave the field entropy, the average information in a source, the bit as its unit, and a result with a sharp consequence: the shortest possible code for a symbol drawn with probability $p$ has length $-\log_2 p$ bits (Shannon 1948). Read that identity backwards and it stops being about codes. A model that assigns probability $p$ to the next token (the next chunk of text it predicts) pays $-\log_2 p$ bits to encode it. A model's loss is a single number measuring how wrong its predictions are, a quantity training drives down; the cross-entropy loss the model is trained to minimize, summed over a corpus, is then exactly the size in bits of that corpus compressed under the model. Lowering the loss and compressing the text harder are the same act. The correspondence is precise rather than poetic: for any distribution there is a prefix code of length $\lceil -\log P\rceil$, and any prefix code defines a distribution, an identity Grünwald calls the most important observation behind the whole theory (Grünwald 2004), and arithmetic coding, which emits fractional-probability symbols as one near-optimal bit stream, realizes it to within a couple of bits of the entropy.

The runnable makes the equivalence concrete: a model that has learned the symbol frequencies compresses to the entropy floor, while an ignorant uniform model wastes bits, and the per-symbol cost of each is just its cross-entropy.

import numpy as np
from collections import Counter

msg = "abracadabra"
freq = Counter(msg); n = len(msg)
p_emp  = {c: freq[c] / n for c in freq}              # a model that learned the frequencies
p_unif = {c: 1 / len(freq) for c in freq}           # an ignorant uniform model

def bits(model):                                     # optimal code length under the model
    return sum(-np.log2(model[c]) for c in msg)

print(f"uniform model:   {bits(p_unif):5.1f} bits  ({bits(p_unif)/n:.2f} bits/symbol)")
print(f"frequency model: {bits(p_emp):5.1f} bits  ({bits(p_emp)/n:.2f} bits/symbol)")
H = -sum(p_emp[c] * np.log2(p_emp[c]) for c in freq)
print(f"entropy (the floor):              {H:.2f} bits/symbol")
print("a better predictor is a smaller file: cross-entropy in bits IS the code length")

This is not a metaphor that AI later adopted; it is the training objective. Delétang et al. drove the point home by running a trained language model as a literal compressor: Chinchilla 70B, trained on text, compresses ImageNet image patches to 43.4% of their raw size, beating PNG at 58.5%, and LibriSpeech audio to 16.4%, beating FLAC at 30.3%, on modalities it was never built for (Delétang et al. 2024). The catch sits in the depth of the claim, and it gets dropped more often than not: those are raw rates that ignore the compressor's own size. Counted in full, with a two-part code, an accounting that charges for the model description and then the data under that model, the LLM is a hopeless net compressor of any small file. So the rigorous statement is narrow and true, training minimizes description length, and the marketing statement, the LLM is a great compressor, holds only if the weights are free.

That two-part accounting is itself a borrowed principle. Rissanen's minimum description length (MDL) principle picks the model $H$ that minimizes $L(H) + L(D \mid H)$, the bits to describe the model plus the bits to describe the data given it (Rissanen 1978), which turns Occam's razor into a computable criterion: the hypothesis that compresses most has learned most. The departure from the source is exact and locatable. The ideal version, measuring description length by Kolmogorov complexity, is provably uncomputable, so practical MDL deliberately scales down to a less expressive but computable model class, accepting that some genuinely regular sequences then become incompressible to it (Grünwald 2004). The engineering kept the principle and gave up the ideal.

A reward error the brain also computes

The next borrowing runs in the rare direction, from algorithm to biology. Sutton formalized temporal-difference learning in 1988 with a single move: assign credit by the difference between two successive predictions rather than between a prediction and the final outcome (Sutton 1988). Structurally the update is the ordinary supervised rule with one substitution, the actual outcome replaced by the model's own next prediction,

$$\Delta w_t = \alpha \, (P_{t+1} - P_t)\, \nabla_w P_t,$$

Here $\alpha$ is the learning rate, $P_t$ and $P_{t+1}$ are the old and next predictions, and $\nabla_w P_t$ is the direction in weight space that would raise the old prediction. The important term is the difference $P_{t+1}-P_t$: learning is driven by surprise between two predictions, not by waiting for the final reward. In the reward setting of Chapter 28 this becomes the reward-prediction error $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ that drives the value update. Those symbols belong to that later chapter and are unpacked there; for a first read only the same difference shape matters here. This formalization came first. Nine years later Schultz, Dayan, and Montague, building directly on it, reported that midbrain dopamine neurons fire in proportion to exactly that quantity: they respond to an unexpected reward, fall silent once a cue reliably predicts the reward, and dip below baseline when a predicted reward is withheld, the signature of a prediction error rather than a reward signal (Schultz et al. 1997). An equation written to make a machine learn turned out to describe a circuit in the brain, which is why this borrowing is worth holding apart from the decorative ones.

It is also where the shared math is still being revised, which is the mark of a live connection rather than a closed analogy. Dabney et al. found that dopamine neurons do not all track the same scalar error: different neurons carry optimistic and pessimistic predictions, so the population encodes a whole distribution of future reward, the biological echo of distributional reinforcement learning (Dabney et al. 2020). The scalar prediction-error story was accurate enough to predict the cells. It was also incomplete, and those same cells keep revising it.

The physics inside diffusion

Physics lent the field a generative algorithm outright. The diffusion models of Chapter 12 were derived, not merely inspired, from non-equilibrium thermodynamics: Sohl-Dickstein et al. take a known forward process that slowly destroys the structure in a data distribution, step by step, into noise, then train a reverse process that restores it, and they build the construction on the physics machinery of Markov diffusion kernels and the Jarzynski equality from statistical mechanics (Sohl-Dickstein et al. 2015). The borrowing reached its departure quickly. The modern recipe keeps the forward-and-reverse mathematics and the training objective and sheds nearly all of the physics: practitioners rarely mention thermodynamics, the reverse pass is a long sequence of learned denoising steps rather than a physical relaxation toward equilibrium, and later work recast the whole thing in the language of scores and differential equations. The thermodynamics was the scaffold that got the building up, not part of the building.

Borrowings that stayed metaphor

The borrowings above became mathematics. Others took a word and left the mechanism behind, and reading them as more than a picture is a common way to overrate the brain analogy. Dual-process psychology is the clean case: the System 1 and System 2 labels, coined by Stanovich and West (Stanovich and West 2000) and popularized by Kahneman (Kahneman 2011), map loosely onto a single fast forward pass versus the deliberate chains of Chapter 30, and the picture organizes the design space well, but a model spending more tokens is not running a second cognitive system, and nothing in the architecture corresponds to the distinction. Evolution is similar from the other side: evolution strategies optimize a model by perturbing its parameters and keeping what scores well, and Salimans et al. are explicit that this is a class of black-box optimization, an alternative to gradient-based reinforcement learning, with nothing depending on living systems (Salimans et al. 2017). Biologically it borrows almost nothing, though as an optimization method it does real work.

Even the neuron is mostly a name. McCulloch and Pitts modeled a nerve cell as a threshold logic unit (McCulloch and Pitts 1943), Rosenblatt made it trainable in the perceptron (Rosenblatt 1958), and Hebb gave the rule that co-active cells strengthen their link (Hebb 1949), but the unit inside a transformer is a cartoon of a biological neuron, and the mechanism the field calls attention shares little with cognitive attention beyond the word. The deeper brain theories the field still watches, predictive coding, where the cortex sends predictions down and only the error up (Rao and Ballard 1999), and Friston's free-energy principle, where a system acts to minimize its own surprise (Friston 2010), remain genuine but unsettled as accounts of artificial systems. The pattern is the same each time: borrow, formalize, diverge. Trust the borrowings that became shared equations, the coding view of training, the dopamine match, the thermodynamic derivation of diffusion, as mechanism. Treat the ones that stayed shared words, neuron, attention, memory, the second system, as pictures that guided design without constraining it.

Further reading