The Infrastructure Before This One

Before generative models became a product category, machine learning already made high-volume decisions in search, advertising, recommendations, fraud detection, forecasting, and many other services. These systems built mature ways to retrieve candidates, manage features, deploy models, monitor drift, and run controlled experiments.

This chapter calls that body of work large-scale predictive ML. It is the closest operational predecessor to the generative stack, not a history of all earlier AI. The comparison answers three questions: which practices transferred, which ones had to change, and which systems remain outside this book's scope.

One request through the older stack

The scale was already substantial. In one Facebook production datacenter studied before the generative-model buildout, recommendation models accounted for more than 79% of AI inference cycles (Gupta et al. 2020). A separate study reported that recommendation consumed more than half of the company's AI training cycles (Acun et al. 2021). These figures describe Facebook's measured workloads, not every datacenter, but they show that production AI was already dominated by systems choosing what to display next.

A large recommender cannot carefully score every item for every request. The usual solution is a funnel:

1. Candidate generation quickly reduces a large catalog to a manageable set.
2. Ranking applies a richer model to that set and orders the results.
3. Logging and experimentation record what was shown and what happened next.

YouTube's 2016 system is a well-documented example: a candidate model reduced millions of videos to hundreds, then a ranking model selected a few dozen for display (Covington et al. 2016). The exact architecture varies by product, but the reason for separating retrieval from ranking is stable. The expensive model only runs on a small candidate set.

Embedding retrieval gives the first stage a simple formula:

$$a_i=e_u^{\mathsf T}e_i, \qquad C_k(u)=\operatorname{TopK}_{i\in\mathcal I}(a_i).$$

The ranker then computes

$$\hat{i}_1,\ldots,\hat{i}_m =\operatorname{Sort}_{i\in C_k(u)} r_\psi(u,i,z_i).$$

Here $u$ is the current user or request, $i$ is an item in catalog $\mathcal I$, and $e_u,e_i\in\mathbb{R}^d$ are $d$-dimensional learned vectors called embeddings. Their dot product $a_i$ is the retrieval score. $C_k(u)$ is the set of $k$ retrieved candidates. The second-stage model $r_\psi$, with parameters $\psi$, uses request data and additional item features $z_i$ to order $m\leq k$ results.

Computing every $a_i$ is still too expensive for a very large catalog. Approximate nearest-neighbor (ANN) search avoids exhaustive scoring by trading some retrieval recall for lower latency and memory traffic. Systems such as FAISS support similarity search at billion-vector scale (Johnson et al. 2021), while HNSW organizes vectors as a navigable proximity graph and is widely implemented in search systems (Malkov and Yashunin 2020).

The figure shows a shared systems pattern, not an identical learning problem. A recommender ranker predicts or optimizes item outcomes. A retrieval-augmented generator uses retrieved evidence as input to an open-ended generation process. The retrieval machinery can transfer even though the final model, objective, and evaluation do not.

What transferred

Three parts of the older stack became foundations for generative systems.

The operational inheritance is especially direct. Google's account of hidden technical debt showed that production failures often come from data dependencies, feedback loops, configuration, and surrounding code rather than from the model alone (Sculley et al. 2015). Uber's Michelangelo platform later standardized the full workflow: shared feature definitions, matching transformations for training and serving, versioned artifacts, deployment, and monitoring (Hermann and Del Balso 2017). TensorFlow Serving supplied versioned model loading, request batching, controlled rollout, and rollback (Olston et al. 2017). The objects changed in generative applications, but the need for lineage and safe rollout did not.

Measurement transferred with an important warning about time. Hohnhold et al. showed that users' future tendency to engage with ads depended on the relevance and landing-page quality of ads they had previously seen. Their long-term model led Google to increase the auction's emphasis on quality and cut mobile-search ad load by 50%, with neutral or positive long-run business impact (Hohnhold et al. 2015). In 2025, OpenAI rolled back a GPT-4o update that had become overly agreeable and wrote that it had focused too much on short-term feedback (OpenAI 2025). Clicks and thumbs-up signals are not the same mechanism. The shared lesson is narrower: immediate feedback can be a poor proxy for the outcome a product needs over time.

Offline improvement is not product improvement

The Netflix Prize remains a compact example of why production systems need more than an offline score. Netflix launched the competition in 2006 and offered one million dollars for a 10% improvement in rating-prediction accuracy. Two algorithms from the 2007 Progress Prize's 107-model ensemble were incorporated into the service. The additional methods in the later Grand Prize solution were not: Netflix reported that their accuracy gain did not justify the engineering work, while the product had also shifted from DVD rating prediction toward helping streaming viewers choose what to watch (Amatriain and Basilico 2012).

The episode separates three deployment gates:

1. Offline quality: does a model improve the chosen test metric?
2. Systems feasibility: can it meet cost, latency, reliability, and maintenance requirements?
3. Product value: does the metric still represent the decision the product must make?

Later chapters apply the same distinction to benchmarks, model serving, and operational evaluation. A better offline number is evidence, not a deployment decision.

Why serving changed

The older operational discipline transferred, but its serving architecture was not a drop-in fit. The useful comparison is between representative workloads, not between all recommenders and all language models.

The first column describes models such as DLRM, which combine sparse categorical embeddings with dense interaction networks (Naumov et al. 2019). Which component dominates depends on the model, batch, and hardware. The second column also contains two regimes: prefill usually has higher arithmetic intensity, while decode at low or moderate batch sizes often moves model weights faster than it can reuse them (Pope et al. 2023).

The per-sequence attention state makes the serving difference concrete. Its approximate size is

$$M_{\mathrm{KV}}\approx 2L H_{\mathrm{KV}}d_h S b.$$

Here $M_{\mathrm{KV}}$ is the key-value cache size in bytes, the factor $2$ counts keys and values, $L$ is the number of transformer layers, $H_{\mathrm{KV}}$ is the number of key/value heads, $d_h$ is each head's width, $S$ is the number of cached tokens, and $b$ is the number of bytes per stored element. The cache grows with every generated token. Grouped-query and multi-query attention reduce $H_{\mathrm{KV}}$, but they do not remove the per-request state.

This growth forced new memory and scheduling mechanisms. PagedAttention maps non-contiguous cache blocks in a way analogous to virtual memory, reducing fragmentation and enabling sharing (Kwon et al. 2023). Iteration-level scheduling can then add and remove requests between decode steps. The older stack still contributes RPC, load balancing, batching, observability, staged deployment, and rollback; LLM serving adds a different state-management problem.

What remains outside this book's scope

Production ML is not a sequence in which every tree model is eventually replaced by a neural network. Gradient-boosted decision trees remain strong for many tabular problems, and XGBoost's success came partly from systems work on sparse data, cache use, and out-of-core execution (Chen and Guestrin 2016). A 2022 benchmark found tree ensembles ahead of tested deep networks on 45 curated medium-size tabular datasets (Grinsztajn et al. 2022).

The boundary is moving, but the evidence is scoped. TabPFN, a transformer pretrained on synthetic tabular tasks, outperformed tuned tree baselines in the authors' evaluations of datasets with at most 10,000 samples and 500 features (Hollmann et al. 2025). The paper also says that behavior beyond those limits needs further study and reports slower per-example inference than CatBoost in a 10,000-row example. Model family is therefore a workload choice, not a progress ladder. This book follows the lifecycle of generative models; tabular prediction, fraud models, forecasting, and classical learning-to-rank remain adjacent production systems rather than hidden chapters of that lifecycle.

Where the stacks are beginning to combine

Recommendation is also adopting generative methods. TIGER assigns each item a short semantic identifier and trains a transformer to decode the identifier of the next item (Rajput et al. 2023). In the evaluated sequential-recommendation pipeline, decoding replaces approximate-nearest-neighbor lookup. A learned codebook, identifier-to-item mapping, and beam search remain outside the model, and the published experiments do not establish web-scale production replacement.

HSTU provides stronger production evidence but supports a narrower conclusion than “the funnel is gone.” Its authors report a 1.5-trillion-parameter generative recommender deployed on several surfaces of a large platform, a 12.4% improvement on one disclosed online metric, and an empirical power-law relation between quality and training compute across three orders of magnitude (Zhai et al. 2024). The same paper retains multiple retrieval generators and separate ranking experiments over candidate sets. Generative models are replacing or augmenting components of the older stack; the evidence does not yet show that one architecture removes every stage.

Closed-loop data is another point of contact. In simulations, Chaney et al. showed that training on interactions produced under earlier recommendations can increase behavioral homogeneity without increasing utility (Chaney et al. 2018). The mechanism is specific: the deployed policy changes which human interactions become observable training data. That is different from inserting model-generated samples into a training corpus, although both cases require engineers to ask who produced the next dataset and under which policy.

The boundary for the rest of the book

The comparison can now be stated without treating every resemblance as an inheritance.

The rest of the book follows a generative capability from compute and data, through training and adaptation, into serving, evaluation, governance, and operation. Earlier predictive systems will reappear when they provide a method or a warning, but they are neighboring infrastructure, not an earlier chapter of the same model lifecycle.

Further reading