Compute Markets and Unit Economics

The whole stack is a way to spend money, and the bill has a shape. The economic map starts with where compute is bought, why training and inference are two different kinds of cost, and when to build a model versus buy one through an API. Its sharpest feedback loop is the claim that inference dominates the lifetime bill: if that claim holds for a deployment, it reaches back up into the architecture and scaling choices of Chapter 5.

An invoice has a shape

Every decision in the earlier chapters resolves, eventually, into a number on an invoice. A parameter count is a training bill. An attention variant becomes a key-value cache, memory rented by the hour, and a sampling budget a column on a serving ledger. The question is one you have probably felt already but not stated outright: given that compute is the scarce input, how is it priced, how does the price split between building a model and running it, and how does that split feed back into what you should build.

Three constraints make the question hard. Compute is rented in a market with volatile prices, so the cost of a FLOP is a moving target, not a constant. Training is paid once and inference is paid forever, so the two costs grow on different clocks. And the cheapest answer is often to buy capability from someone else, which means the build decision competes against a falling market price you do not control. The shape of the invoice, then, is set by two cost structures running on two clocks, by a scissors opening between them, and by a single break-even that decides which side of the market you sit on.

Two cost structures, two clocks

The organizing fact is that an AI system has two cost structures, not one, and they behave differently enough that conflating them produces wrong decisions.

Training is a capital expense (CapEx): a capital-style, one-time spend that is large, lumpy, and paid before any value is returned. You commit a cluster for weeks, spend a fixed sum, and get a set of weights. The spend is dominated by the final run plus the research and ablation runs that found the recipe. Per Epoch AI's analysis of roughly 45 frontier models, the amortized hardware and energy cost of the final training run has grown about 2.4 times per year since 2016, with hardware accounting for the largest single share, R&D staff a comparable share, and energy a small remainder (Cottier et al. 2024). The headline figures are dated and directional: GPT-4 around 40 million dollars, Gemini Ultra around 30 million dollars, and, on that trend, the largest runs crossing a billion dollars around 2027 (Cottier et al. 2024).

Inference is an operating expense (OpEx): an operating cost paid a little at a time on every request. It is small per request and paid every time the model is called, so it scales with usage rather than with the training budget. A useful first-order model of its unit cost is

$$\text{cost per token} \approx \frac{\text{accelerator } \$/\text{hour}} {\text{throughput (tokens/hour)} \times \text{utilization}}$$

which says the three levers on inference cost are the rented price of the accelerator, the throughput the serving stack extracts from it, and how fully the hardware is kept busy. The serving chapters in Part V are, read through this equation, an attack on the denominator: Chapter 32 raises utilization by batching, while Chapter 33 and Chapter 34 raise throughput. Architecture choices such as the attention variants of Chapter 9 shrink the per-token work from the model side.

The two cost structures decompose into different drivers, and Figure 76.3 lays them side by side: training resolves into the additive parts of the final run, while inference resolves into a ratio whose denominator is the target of the Part V serving optimizations.

The compute market sets the numerator. Accelerators are rented by the hour across hyperscalers and a tier of specialist GPU clouds, with three price levels: reserved capacity at the lowest committed rate, on-demand at a premium for flexibility, and spot at the deepest discount in exchange for preemption. The physical input is not only the accelerator. Data-center power, cooling, interconnect, region, and utilization all enter the hourly price, which is why Chapter 68 treats energy as part of AI infrastructure rather than external scenery (International Energy Agency 2025). The directional fact matters more than any single quote: the price of a delivered FLOP changes over time, and a serving plan written against last year's price can be wrong by a margin that compounds.

The scissors: rising training, falling inference

The two clocks do not merely tick at different rates. They often run in opposite directions, and the gap they open is the most consequential economic shift in this part of the stack: the build-versus-buy default has inverted over the life of the field.

Early on, capability lived only inside a few labs, so building was the only way to get a frontier model and buying was not an option. As hosted APIs matured, buying became possible, and then a second curve took over: the price of a token of inference began falling quickly through software efficiency, architectural change, and competition. Du's 2026 pricing study reports an approximately 600-fold decline in token prices from 2020 to 2026, with economy and mid-tier models falling faster than Moore's Law while flagship reasoning models preserve a premium (Du 2026). The release of strong open-weight models, DeepSeek-V3 among them, accelerated this by giving every closed provider a low-priced reference point to justify against (DeepSeek-AI 2024).

Drag the decay parameter to see how fast the per-token price collapses, the lower blade of the scissors that the build decision must race against.

Set the two curves together and the scissors is plain. Training cost rises about 2.4 times per year at the frontier (Cottier et al. 2024), while many served-token prices fall rapidly as the inference market commoditizes (Du 2026). The widening gap between them is why the default answer drifted from build to buy. Building a frontier model from scratch competes against a market price that may be lower by the time the system is ready, so building only pays when something the market cannot sell you is at stake: proprietary data that cannot leave your boundary, a latency or availability floor the API cannot meet, a customization the hosted model will not accept, or an inference volume large enough that self-hosting beats the per-token markup.

The arithmetic of building has itself moved. DeepSeek-V3, a 671-billion parameter mixture-of-experts model, reports a final training run of about 2.788 million H800 GPU-hours, roughly 5.6 million dollars at an assumed two dollars per GPU-hour (DeepSeek-AI 2024). The number is striking, and the report is careful to say what it excludes: the prior research, the architecture and data ablations, and everything that is not the one final run. The lesson for a build decision is that the headline training number is the visible tip; the research that found the recipe is the submerged cost, and it does not appear on the run's invoice.

Where the knees are

Each economic choice is a balance with a knee, a point past which the cheaper option flips.

• Capital versus operating cost. Spending more to train, by over-training a smaller model past Chinchilla-optimal in the sense of Chapter 5, lowers every future inference bill (Sardana et al. 2024). The trade pays only if the lifetime token volume is large enough to amortize the extra training spend, so the right point depends on a usage forecast you make before you have the usage.
• Build versus buy. Buying through an API converts a capital expense into a per-token operating expense and rides the falling market price, at the cost of control over data, latency, and customization. Building converts it back into a capital expense plus a fixed serving footprint, which wins only past a break-even volume or where a non-price constraint forces it.
• Reserved versus spot capacity. Reserved compute is cheaper per hour and predictable, which fits steady production serving. Spot is cheaper still but can be preempted, which fits interruptible work: batch inference, checkpointed training, hyperparameter sweeps. Matching the workload's interruption tolerance to the pricing tier is most of the savings.
• Quality versus unit cost. A larger or reasoning-heavy model answers better and costs more per request. The economically right model is the smallest one that clears the task's quality bar, which is why routing a workload across a ladder of model sizes often beats sending everything to the largest.

The capital layer

The unit economics above assume the accelerators exist. Zoom out one level, to how they are paid for, and a second economy appears whose accounting choices swing the numbers this chapter reports. It starts with a deceptively technical question: over how many years does a GPU wear out? A training accelerator costs the same cash whenever it is bought, but the reported cost is that cash spread across its useful life, so the choice of useful life sets how much cost lands in any given year. Through 2022 and 2023 the major hyperscalers extended server useful lives from four years to six, each citing efficiency gains, and each extension lifted reported income by roughly three billion dollars a year (Computer Weekly 2023). The counter-move is as telling: Amazon reversed course in 2025, shortening a subset of servers back to five years and citing "the increased pace of technology development, particularly in the area of artificial intelligence" (Amazon 2025). Schedules move in both directions because the underlying fact is contested. The bear case, argued publicly and sharply in late 2025, is that an annual accelerator cadence makes the economic life two to three years even if the silicon runs for six, so stretched schedules understate depreciation and overstate earnings across the industry (CNBC 2025). The vendor's rebuttal points to still-productive prior-generation parts rebooking near their original price (CNBC 2025). The runnable makes the mechanism concrete: the same cash spent buys very different reported costs.

fleet_capex = 60e9          # one year of accelerator purchases, USD
horizon = 6
for life in [3, 6]:         # the two contested straight-line schedules
    annual = fleet_capex / life
    charges = [annual if y < life else 0.0 for y in range(horizon)]
    print(f"{life}-yr life: ${annual/1e9:.0f}B/yr  ->  "
          f"{[f'{c/1e9:.0f}' for c in charges]}")
gap = fleet_capex/3 - fleet_capex/6
print(f"reported-cost swing from the schedule choice alone: ${gap/1e9:.0f}B/yr")
print("identical cash left the building in year 0 either way")

Depreciation is contested because the second development below it is real: debt has entered the buildout at a scale operating cash flow no longer covers. One analysis put global datacenter spend near three trillion dollars through 2028 against roughly half that in hyperscaler cash flow, leaving a financing gap on the order of a trillion and a half, much of it expected from asset-backed private credit (Morgan Stanley Research 2025), and central-bank researchers documented big-tech bond issuance passing one hundred billion dollars in 2025, with much of the borrowing moved into off-balance-sheet special-purpose vehicles (Eren et al. 2026). The instruments are novel: GPU-collateralized loans that lend against a chip's contracted revenue rather than its resale value (CoreWeave 2023), and datacenter joint ventures in which a credit fund owns most of a campus and the hyperscaler leases the capacity back, keeping the debt off its own books (Meta 2025). And the money moves in circles that invite the comparison to the telecom vendor-financing of the dot-com era: a chip vendor investing tens of billions in a model lab that commits to buying its chips, a supplier granting a customer equity warrants for placing an order, a cloud provider holding a large stake in the lab that has contracted to spend a quarter-trillion dollars on its cloud (NVIDIA 2025; AMD 2025; Microsoft 2025). Skeptics read these loops as demand manufacturing its own evidence; the counterargument points to served-inference volume growing roughly twentyfold year over year, which is hard to round-trip (Alphabet 2025). The sharper form of the bear case is therefore not about volume but about price per token and margin.

The exposure has become macroeconomic. Information-processing investment, a category wider than datacenters but dominated by them, accounted for the large majority of measured US GDP growth in the first half of 2025 (Furman 2025), and AI-related investment as a share of GDP now sits above the late-1990s telecom peak while remaining below the railroad and electrification buildouts (Goldman Sachs 2026). The historical rhyme worth carrying is Odlyzko's on the railway mania: a genuinely transformative technology and a ruinous capital-destruction episode are not mutually exclusive, and the demand extrapolations that justified the overbuild were, in hindsight, ignoring available quantitative evidence (Odlyzko 2010).

Running the two numbers

The economics become operational through two small calculations a team actually runs, one for each clock.

The first is the inference unit cost above. To turn it into a number you need the accelerator's hourly price from the compute market, the serving stack's measured throughput in tokens per hour, and the realized utilization. The same equation explains why the serving optimizations in Part V pay for themselves directly: doubling throughput or utilization halves the cost per token, and the per-token cost is what every downstream margin is computed against.

The second is the build-versus-buy break-even. Here $p$ is the API price per token, $c$ is your own marginal cost per token from the equation above, and $F$ is the fixed cost of standing up and reserving your own serving capacity. Buying costs $p \times V$ for a volume $V$; building costs $F + c \times V$. Building wins once

$$V > \frac{F}{p - c}$$

Read the inequality as a payback test. The numerator $F$ is the one-time commitment you must recover; the denominator $p-c$ is the savings on each token after you stop buying from the API and serve it yourself. The threshold exists as a positive number only when your marginal cost $c$ is below the API price $p$. The break-even moves against you over time because $p$ falls with the market, so a build that pencils out today must be re-checked against next year's lower API price before the capital is committed.

Try changing the annual price decline or the fixed cost below and watch the break-even volume climb as the API price falls toward your marginal cost.

import numpy as np

F = 2_000_000.0   # fixed cost to stand up and reserve serving ($)
c = 0.20          # your marginal cost per million tokens ($)
p0 = 2.00         # API price per million tokens today ($)
decline = 0.40    # fraction the API price falls each year

for year in range(4):
    p = p0 * (1 - decline) ** year
    if p > c:
        breakeven = F / (p - c)   # millions of tokens to justify building
        print(f"year {year}: p=${p:5.2f}/Mtok  break-even = {breakeven:,.0f} Mtok")
    else:
        print(f"year {year}: p=${p:5.2f}/Mtok  build never pays (p <= c)")

The full decision is not one inequality but a short branching path, drawn in Figure 76.6: a non-price constraint short-circuits straight to build, the volume threshold decides the rest, and a falling $p$ forces a recheck before any capital is spent.

The failure modes are economic, not numerical. Building against a falling buy price is the classic one: a self-hosted plan justified at this year's API price can be underwater a year later, and the capital is already spent. Provisioning steady reserved capacity for spiky demand wastes the trough; serving spiky demand on pure on-demand or spot pays a premium or risks preemption at the peak. And forecasting inference volume too low at training time under-trains a model that will be served far more than expected, leaving a permanent per-token cost on the table that no serving optimization fully recovers.

Further reading