Borrowed Ideas: What AI Took From Other Sciences

AI is built from ideas that crossed disciplinary boundaries. Information theory gave machine learning a language for prediction and coding. Neuroscience inspired simplified computational units. Statistical physics supplied tools for reasoning about large systems and stochastic processes. Psychology contributed names for different styles of reasoning.

These connections are not all equally strong. A shared equation may express an exact identity, a useful model of another system, or only a mathematical resemblance. A shared name may preserve part of an older idea without preserving its mechanism. It helps to distinguish four relationships:

Four questions make the strength of each connection explicit:

1. Mapping: What variables, operations, and assumptions correspond?
2. Preservation: Which equations or constraints remain valid?
3. Transfer: Does the source idea predict behavior or enable an intervention in the new setting?
4. Boundary: Where does the correspondence stop?

A matching equation without a clear mapping is only a formal resemblance. Even a model that predicts observations does not by itself prove that two systems share a physical mechanism.

Formal identity: prediction loss and codelength

Suppose an autoregressive model assigns probability $q(x_t\mid x_{<t})$ to token $x_t$ after seeing the preceding tokens $x_{<t}$, where $t$ is the token's position. The probability it assigns to the whole sequence is

$$q(x_{1:n})=\prod_{t=1}^{n}q(x_t\mid x_{<t}).$$

Its ideal codelength, measured in bits, is therefore

$$L_q(x_{1:n})=-\log_2 q(x_{1:n}) =\sum_{t=1}^{n}-\log_2 q(x_t\mid x_{<t}).$$

Here $x_{1:n}$ is the sequence of $n$ tokens, $q$ is the model, and $L_q$ is the ideal number of bits charged by that model. The sum on the right is also the sequence's base-2 negative log-likelihood. This is an exact formal identity, not an analogy (Shannon 1948).

The word ideal matters. A binary codeword has an integer length, so a symbol with probability $q(x)$ cannot generally receive a literal codeword of exactly $-\log_2 q(x)$ bits. A Shannon code can use $\lceil-\log_2 q(x)\rceil$ bits per symbol. An entropy coder can do better over an entire sequence: arithmetic coding, which turns sequence probabilities into one near-optimal bit stream, can approach the ideal sequence length with small coding and framing overhead (Grünwald 2004). The tokenizer and probability model must also be known to both encoder and decoder.

The following runnable compares two fixed probability models. It computes ideal model-based codelengths; it does not create a file or charge for transmitting the models.

from math import log2

message = "abracadabra"
informed = {"a": 0.4, "b": 0.2, "r": 0.2, "c": 0.1, "d": 0.1}
uniform = {symbol: 0.2 for symbol in informed}

def ideal_bits(model):
    return sum(-log2(model[symbol]) for symbol in message)

for name, model in [("uniform", uniform), ("informed", informed)]:
    total = ideal_bits(model)
    print(f"{name:8s}: {total:5.2f} ideal bits ({total / len(message):.2f} per symbol)")

The better predictions cost fewer ideal bits. That statement applies directly to autoregressive training with log loss. It does not mean that every machine- learning objective is a compressor, or that the resulting stored file has exactly that size.

Delétang et al. demonstrated the operational connection by coupling trained models to a lossless entropy coder (Delétang et al. 2024). In the final paper's table, Chinchilla 70B encoded the study's 2,048-byte image and audio samples at 48.0% and 21.0% of their raw sizes, below the corresponding chunked PNG and FLAC rates of 61.7% and 30.3%. Those rates assume the model weights are already shared. Charging for a 70-billion-parameter model reverses the comparison for a small dataset.

That distinction leads to the minimum description length (MDL) principle. In its simple two-part form, choose a model $M$ that minimizes

$$L(M)+L(D\mid M),$$

where $L(M)$ is the cost of describing the model and $L(D\mid M)$ is the cost of describing data $D$ using it (Rissanen 1978). This is a two-part code, an accounting that charges for the model description before crediting the model's compression of the data. Ordinary maximum-likelihood training minimizes the second term for a fixed model family; it does not normally charge for architecture, learned weights, or their precision. Practical MDL also depends on the candidate models and coding scheme because the ideal Kolmogorov-complexity version is not computable (Grünwald 2004).

Computational correspondence: TD error and dopamine

Ideas also travel in the other direction. Temporal-difference (TD) learning was developed as a machine-learning method, then became a quantitative model for some neural responses (Sutton 1988). In one-step value learning, the TD error is given by this formula:

$$\delta_t=r_{t+1}+\gamma V_w(s_{t+1})-V_w(s_t),$$

and the parameters are updated by

$$w_{t+1}=w_t+\alpha\delta_t\nabla_w V_w(s_t).$$

Here $s_t$ is the current state, $r_{t+1}$ is the next reward, $V_w(s)$ is the estimated discounted return from state $s$, $\gamma$ discounts future rewards, $\alpha$ is the learning rate, and $w$ contains the value function's parameters. The next-state estimate supplies a bootstrapped target; in this semi-gradient update, that target is treated as fixed while differentiating the current estimate.

Schultz, Dayan, and Montague reported that phasic activity in recorded midbrain dopamine neurons showed several patterns predicted by a TD reward-prediction error (Schultz et al. 1997). An unexpected reward produced a response. After a cue reliably predicted that reward, the response shifted toward the cue. If the predicted reward was omitted, activity briefly dipped around the expected time. This was a strong, testable correspondence between a computational model and biological measurements.

The boundary is just as important. Those observations do not show that dopamine implements the complete TD algorithm, and dopamine activity is heterogeneous. Dabney et al. later measured asymmetric responses to positive and negative errors across mouse dopamine neurons, evidence consistent with a population code over a distribution of discounted returns (Dabney et al. 2020). The result refines the correspondence; it does not turn the model into a complete account of the circuit.

Mathematical imports without physical mechanisms

Diffusion models inherited a construction from stochastic physics. Sohl-Dickstein et al. introduced a fixed Markov process that gradually maps data to a simple noise distribution and a learned reverse process that maps noise back toward data (Sohl-Dickstein et al. 2015). A common Gaussian forward step has the following formula:

$$q(x_t\mid x_{t-1}) =\mathcal{N}\!\left(\sqrt{1-\beta_t}\,x_{t-1},\,\beta_t I\right),$$

while the learned reverse step is modeled as

$$p_\theta(x_{t-1}\mid x_t).$$

Here $x_t$ is the sample after step $t$, $\beta_t$ sets the amount of added Gaussian noise, $I$ is the identity covariance matrix, $q$ is the fixed forward process, and $p_\theta$ is the learned reverse process. Later formulations use denoising, score matching, or stochastic differential equations, but retain the central idea of learning to reverse a controlled corruption process; Chapter 12 develops those versions.

This is genuine mathematical and historical inheritance. It is not a claim that an image generator contains heat, conserves physical energy, or relaxes like a material system. The stochastic construction transferred. The physical medium and causal mechanism did not.

Evolution strategies make a similar separation. A modern version samples parameter perturbations, evaluates the resulting policies, and applies a fitness-weighted parameter update (Salimans et al. 2017). The algorithm retains the abstract pattern of variation and selection. Its correctness as an optimizer does not depend on reproducing genes, organisms, or ecological competition.

What is contested: is emergence a phase transition?

Statistical mechanics has produced real phase-transition results for specified learning models (Seung et al. 1992). It is tempting to use the same language when an ability appears abruptly as language models grow. Wei et al. called an ability emergent when it was absent in smaller models but present in larger ones under the available evaluation (Wei et al. 2022). That describes an observed benchmark curve; it does not establish a physical phase transition.

Measurement can create a cliff. If a model's probability of the correct answer rises smoothly from 0.2 to 0.4, exact-match accuracy may remain at zero and then jump when the top-ranked answer changes. A continuous score can reveal the progress that the thresholded metric hides. Schaeffer et al. showed that nonlinear or discontinuous metrics can explain some reported cases and can even produce apparent emergence in other model families (Schaeffer et al. 2023).

A stronger phase-transition claim would need more than an abrupt benchmark: dense measurements across scale, results robust to the metric, a clearly defined order parameter, and scaling behavior expected near a critical point. Until those tests are supplied for a particular case, “phase transition” is a hypothesis or analogy, not an established mechanism. Metric artifacts can explain some cliffs without proving that every capability always improves smoothly.

Heuristic analogy: what the names preserve

Loose analogies can still guide design if their limits remain visible.

The mistake is not using an analogy. The mistake is allowing its familiarity to supply evidence that has not been measured. For any borrowed idea, ask what maps, what mathematical structure survives, what new prediction transfers, and where the connection fails. The answer tells you whether you are using an identity, a scientific model, an imported tool, or a helpful picture.

Further reading