The Infrastructure Before This One

The generative stack this book takes apart is not the first AI infrastructure. For fifteen years before a language model wrote its first paragraph, machine learning already ran the world's feeds, search results, and ad auctions at planetary scale, and it did so with its own hardware fleets, its own serving tiers, and its own hard-won operational discipline. This chapter puts that incumbent next to the book's subject and asks three questions of it. What did the generative stack inherit? What could it not reuse, and why? And where are the two stacks now converging? By the end, a reader can place the rest of this book against the AI that was already infrastructure, and can recognize which of the "new" problems in later parts are old problems wearing new names.

The largest AI system nobody calls AI

Start with the incumbent's size, because it reframes what "AI infrastructure" meant until very recently. In 2020, before the language-model buildout, Facebook reported that recommendation models consumed up to 79% of the AI inference cycles in its production datacenters (Gupta et al. 2020), and a companion study put their share of training cycles above 50% (Acun et al. 2021). The largest AI fleets on earth were not running chatbots or image generators. They were deciding, billions of times a day, which item a person would see next.

The system doing that deciding has a canonical shape, fixed in the industry's vocabulary by YouTube's 2016 description of its recommender (Covington et al. 2016). The corpus is too large to score every item for every request, so the work splits into two stages. A cheap first stage, candidate generation, narrows millions of items to a few hundred plausible ones. An expensive second stage, ranking, scores those few hundred carefully and orders the final screen. Both stages lean on the same primitive: the embedding, a learned vector that represents a user or an item so that geometric closeness stands in for behavioral affinity. Score by dot product, retrieve by nearest neighbor, and the two-stage funnel turns an impossible scoring problem into two affordable ones.

The funnel was only half the incumbent's maturity. The other half was operational. Google's 2013 report on its ad click-through predictor reads today like a systems paper that happens to contain a model: online learning over billions of sparse features, yes, but also calibration, feature lifecycle management, memory accounting, and dashboards for watching the model drift (McMahan et al. 2013). The discipline now sold as MLOps was worked out in these trenches a decade before the term reached the generative stack.

One episode from the incumbent's history earns retelling because the whole book keeps meeting its lesson. In 2006 Netflix offered a million dollars to any team that could improve its rating predictor's accuracy by ten percent, and the three-year contest that followed pulled much of the field into recommendation research. The winning entry was an ensemble of hundreds of blended models. Netflix never deployed it. The company's engineers later wrote that the measured accuracy gain "did not seem to justify the engineering effort needed to bring them into a production environment," and that the business had meanwhile moved from predicting ratings to ranking what a member would actually watch; only two component algorithms from an earlier milestone ever shipped (Amatriain and Basilico 2012). An offline metric had been optimized brilliantly, and the product did not want it. That gap between the number a benchmark rewards and the behavior a deployment needs returns in this book as a central theme of Chapter 47 and everything after it.

The workhorse that deep learning had to beat

A second correction to the mental picture: the incumbent was never uniformly neural. A large share of production machine learning, then and now, runs on gradient-boosted decision trees, an ensemble method that builds hundreds of shallow decision trees in sequence, each one trained to correct the residual errors of the trees before it. The method's flagship implementation, XGBoost, is revealing about what actually wins in practice: its paper spends most of its pages on systems concerns, cache-aware layout, sparsity handling, out-of-core computation, and notes in passing that 17 of the 29 published Kaggle winning solutions in 2015 used it (Chen and Guestrin 2016). Its ranking-specialized ancestor, LambdaMART, carried web search relevance for a decade (Burges 2010).

Trees were not a placeholder waiting for deep learning to arrive. When Airbnb replaced its hand-written search scoring function with a boosted-tree model, the company described the gain as one of the largest single improvements in bookings in its history; the subsequent migration from trees to neural networks failed twice before it succeeded, and succeeded only after the team stopped trying to out-engineer the trees feature by feature (Haldar et al. 2019). The pattern has a measured explanation. A careful 2022 benchmark found that on typical tabular data, the rows-and-columns business records that most production models consume, tree ensembles still beat deep networks, and traced the advantage to inductive bias: trees are indifferent to uninformative features and fit the abrupt, axis-aligned patterns that tabular targets actually have (Grinsztajn et al. 2022). The frontier is moving even here, and in a familiar direction: TabPFN, a transformer pretrained on millions of synthetic tabular tasks, now beats tuned tree ensembles on small datasets by reading the whole training table as context (Hollmann et al. 2025). But that result holds below roughly ten thousand rows. The high-volume ranking, fraud, and credit tables of the operating economy remain, as of this writing, tree territory.

The point of this section is scope. "AI as infrastructure" did not begin with the transformer, and it does not end at the transformer's edges. A working layer of the economy runs on models this book will not take apart. What the book does take apart sits beside that layer, borrows from it, and is slowly merging with it.

What the new stack inherited

Three inheritances run from the incumbent into the parts ahead, and naming them now will make later chapters feel less like invention from nothing.

The first is retrieval. The candidate-generation stage of the funnel matured into a precise recipe: train two embedding towers, one for queries and one for items, so that dot products rank well; correct for the biases of training on your own logged traffic (Yi et al. 2019); index the item embeddings in an approximate nearest-neighbor structure, a data structure that trades exact search for sublinear lookup, using libraries and graph indexes built for billion-item catalogs (Johnson et al. 2019; Malkov and Yashunin 2018). Facebook's 2020 account of embedding-based retrieval in its search engine reads, in hindsight, like a checklist of what retrieval-augmented generation teams would rediscover four years later: hybrid dense-plus-keyword retrieval, hard-negative mining, the tension between index recall and ranker quality (Huang et al. 2020). HNSW, the graph index inside most of today's vector databases, was published eight years before anyone said "vector database." When Chapter 44 and Chapter 45 develop this machinery for language models, they are extending a production lineage, not starting one.

The runnable below compresses the whole retrieval inheritance into a dozen lines: factorize a matrix of user-item interactions into two towers of embeddings, then serve a user with one matrix-vector product and an argsort. That serving step costing almost nothing is the entire reason billion-item catalogs, and now million-chunk document stores, can be searched interactively.

import numpy as np
rng = np.random.default_rng(0)
n_users, n_items, d = 200, 1000, 16
U0, V0 = rng.normal(size=(n_users, d)), rng.normal(size=(n_items, d))
liked = (U0 @ V0.T + rng.normal(scale=2.0, size=(n_users, n_items))) > 2.0   # synthetic clicks
U, V = rng.normal(scale=0.1, size=(n_users, d)), rng.normal(scale=0.1, size=(n_items, d))
for _ in range(300):                          # two towers, logistic loss, plain SGD
    err = liked - 1 / (1 + np.exp(-(U @ V.T)))
    U, V = U + 0.5 * err @ V / n_items, V + 0.5 * err.T @ U / n_users
u = 7
scores = U[u] @ V.T                           # retrieval: one matrix-vector product
top10 = np.argsort(scores)[::-1][:10]         # ...plus an argsort
print("hit rate in this user's top-10:", liked[u, top10].mean())
print("hit rate of random items:      ", liked[u].mean().round(3))

The second inheritance is the operational shell. The incumbent learned early that a model is the small box in the system diagram, surrounded by data pipelines, configuration, and feedback loops whose failure modes are worse than the model's (Sculley et al. 2015). Out of that lesson came the feature store, a service that computes each input feature once and serves the identical value to training and to production, because the two silently diverging, training/serving skew, was among the most common ways a model that evaluated well behaved badly (Hermann and Del Balso 2017). It also produced the serving tier itself: versioned models loaded behind an atomic pointer swap, request batching, canaried rollouts (Olston et al. 2017). Readers who reach Chapter 89 and Chapter 46 will meet both patterns transposed: the prompt and context pipeline is the new feature pipeline, and it drifts between evaluation and production in exactly the way feature stores were built to prevent.

The third inheritance is measurement, and here the incumbent remains ahead. Two decades of high-volume A/B testing produced a discipline of overall evaluation criteria, guardrail metrics, and rigorous variance accounting that is still the reference standard for knowing whether a change helped (Kohavi et al. 2020). Its sharpest lesson concerns time horizons. Google showed in 2015 that optimizing short-term click-through quietly teaches users to ignore ads, so that the metric being maximized was borrowing against the future; the fix was to measure long-term user learning and deliberately trade short-term revenue away (Hohnhold et al. 2015). Ten years later, a chat assistant tuned on thumbs-up feedback became so agreeable it had to be rolled back, and the vendor's post-mortem said it had "focused too much on short-term feedback" (OpenAI 2025). Preference-trained models drift toward flattery for reasons Chapter 19 will make precise (Sharma et al. 2024), but the shape of the failure, a cheap proxy metric optimized into user harm, was documented by the incumbent a decade earlier. The new stack is relearning it with a larger model in the loop.

What did not transfer

Given all that inheritance, a reader might expect the generative stack to have reused the incumbent's serving tier outright. It could not, and the reason is a lesson in how workload shape dictates infrastructure.

A production recommender model is, computationally, an inversion of a transformer. Meta's deep learning recommendation model is dominated by embedding tables that can reach terabytes, from which each request randomly fetches a few hundred rows; the dense arithmetic stitched on top is small (Naumov et al. 2019). The binding resources are memory capacity and random-access lookup rate, not arithmetic. A language model at inference is the opposite: nearly all cost is dense matrix arithmetic, and each generated token requires streaming the model's weights and the conversation's key-value cache through memory again (Pope et al. 2022). One workload is a sparse lookup machine with a little math attached; the other is a dense math machine with a bandwidth problem.

Statefulness split the two tiers just as cleanly. A recommender request is stateless: user state lives in the feature store, so any replica behind the load balancer can serve any request, and the incumbent's whole serving pattern leans on that freedom. An in-flight generation is stateful, because the KV cache accumulates on the accelerator token by token, and it is why LLM serving had to invent request-level memory management, paging, and continuous batching rather than borrow the stateless tier (Kwon et al. 2023). Latency budgets inverted too: the recommender spends a fixed scoring cost per request and invests its budget in funnel width, while generation is open-ended, priced per output token, and split into the time-to-first-token and per-token cadence that Chapter 31 dissects.

The convergence

The two stacks are now flowing into each other from both directions, which is the strongest evidence that they were always one field with two workload shapes.

Generative technique is entering recommendation. Google's TIGER reframed retrieval itself as generation: quantize each item's content embedding into a short code called a semantic ID, then train a sequence model to emit the next item's code directly, so the approximate-nearest-neighbor index disappears into the model's weights (Rajput et al. 2023). Meta pushed the recipe to production scale, reporting trillion-parameter generative recommenders that beat its classical stack by double digits in online experiments and, notably, that improve as a power law of training compute, the same scaling behavior Chapter 5 builds this book on (Zhai et al. 2024). The incumbent, having lent the generative stack its funnel, is now borrowing back the generative stack's training recipe.

Traffic flows the other way as well, and it carries warnings. Recommender research documented years ago that a system trained on traffic generated under its own previous policy drifts toward homogeneity, a feedback loop the field learned to detect and dampen (Chaney et al. 2018); the generative analog, models training on a web increasingly written by models, is an open problem that Chapter 70 confronts with less accumulated scar tissue. And the argument that engagement-optimized recommenders were the first misaligned objective deployed at planetary scale (Stray et al. 2021) hangs over every preference-optimization chapter in Part III: the incumbent ran the experiment of pointing a powerful optimizer at a cheap proxy for human satisfaction first, at civilizational sample size.

The boundary this chapter draws

This chapter fixes the book's scope claim. What follows is the lifecycle of a generative capability: compute into base model, base model into adapted behavior, behavior into a served, evaluated, governed system. The incumbent infrastructure of ranking, recommendation, and tabular prediction is not that lifecycle, and this book does not retell its playbooks. It appears from here on the way it appeared in this chapter: as the source of an inherited pattern, as the operator of the fleet next door, and as the field that already paid for a lesson the generative stack is about to buy again. Where a later chapter meets one of those moments, retrieval funnels in Chapter 44, deployment gates in Chapter 89, proxy-metric drift in Chapter 53, it will point back rather than pretend a fresh start. Judged by the book's recurring lens, the incumbent's standing is uneven: mature in efficiency, far ahead in measured trust, and now importing capability. The generative stack is the mirror image, and the rest of this book is about what it costs to close that gap.

Further reading