Compute Markets and Unit Economics

The whole stack is a way to spend money, but an invoice is not yet an economic model. This chapter asks where compute is bought, why training and inference are two different kinds of cost, and when to build a model versus buy one through an API. It also tests, rather than assumes, the claim that inference dominates the lifetime bill.

Every answer depends on a declared workload, service objective, time horizon, and accounting boundary. A low token price can still produce an expensive product if the model needs long outputs, repeated attempts, tool calls, or human review. A fast self-hosted model can still be uneconomic if enough idle capacity must be reserved for rare peaks. The useful unit is therefore the cost per accepted result, measured for alternatives that satisfy the same operating contract.

Separate three decisions

“Build or buy” compresses three decisions that have different costs and exit paths:

1. Create or acquire capability. A team can train from scratch, continue training or fine-tune an existing model, use open weights unchanged, or buy access to a hosted model.
2. Operate or outsource serving. Open weights can run on owned or rented capacity, a managed endpoint can host them, and a provider API can bundle model and serving. Training a model is not the same decision as self-hosting one.
3. Procure capacity. Owned capacity, reserved capacity, on-demand capacity, and interruptible spot capacity exchange price for commitment, capacity assurance, and interruption risk.

Screen feasibility before comparing prices. Two candidates are not economically comparable if one misses the quality threshold, tail-latency objective, availability target, privacy boundary, security controls, licensing terms, or required human review. Treat tail latency as part of the contract, not as a note beside average throughput. A provider can also offer private networking, regional processing, dedicated capacity, or managed open weights, so a non-price constraint does not automatically imply training from scratch.

Chapter 75 supplied the versioned component contracts. The cost record attaches a ledger and a forecast to the exact model, runtime, configuration, policy, workload, and evidence tuple that passed those contracts.

Build the complete ledger

Choose an evaluation interval, such as a month, and keep cash and resource units explicit. A useful all-in ledger is

$$\begin{aligned} C_{\mathrm{total}} &= C_{\mathrm{build}} \\ &\quad + \sum_{t=1}^{T}\bigl( C_{\mathrm{run},t} + C_{\mathrm{people},t} \\ &\qquad + C_{\mathrm{data},t} + C_{\mathrm{network},t} \\ &\qquad + C_{\mathrm{failure},t}\bigr), \\ u_{\mathrm{accept}} &= \frac{C_{\mathrm{total}}}{N_{\mathrm{accept}}}. \end{aligned}$$

Each symbol makes the accounting boundary explicit:

• $C_{\mathrm{total}}$ means all cost assigned to the evaluated system over the horizon, while $C_{\mathrm{build}}$ means experiments, model adaptation, integration, evaluation, and launch work incurred before routine operation.
• $t$ denotes one evaluation interval and $T$ the number of intervals in the declared horizon. The sum adds the recurring terms across those intervals.
• $C_{\mathrm{run},t}$ covers compute, storage, licenses, energy, and managed services in interval $t$; $C_{\mathrm{people},t}$ covers engineering, operations, support, security, and review.
• $C_{\mathrm{data},t}$ covers data acquisition, labeling, retention, and deletion; $C_{\mathrm{network},t}$ covers transfer, private connectivity, and egress; $C_{\mathrm{failure},t}$ covers failed and retried work, incident response, credits, and waste.
• $N_{\mathrm{accept}}$ counts results that meet the declared quality, latency, policy, and completion criteria. The resulting $u_{\mathrm{accept}}$ is the all-in cost per accepted result.

The ledger needs a written shared-cost allocation rule. A gateway, evaluation team, observability system, or reserved cluster may serve several products. Allocating all of it to the newest product exaggerates that product's cost; allocating none of it hides a real dependency. Record the allocation driver, such as accelerator-hours, requests, storage bytes, or staff time, and show the result both with and without disputed allocations.

The denominator matters just as much. Raw requests count malformed calls, policy denials, retries, and answers that fail the task. Raw output tokens can reward verbosity. Define an accepted result using the same evaluation and service contract used for release. For exploratory systems, report several denominators instead of forcing one: cost per request, accepted result, resolved case, and unit of user value.

Measure serving cost from realized work

For rented or owned accelerator capacity, isolate the accelerator portion of the unit cost as

$$C_{\mathrm{accel}} = r_{\mathrm{eff}} H_{\mathrm{billed}}, \qquad u_{\mathrm{accel}} = \frac{C_{\mathrm{accel}}}{N_{\mathrm{accept}}} = \frac{r_{\mathrm{eff}}H_{\mathrm{billed}}} {N_{\mathrm{accept}}}.$$

Here, $C_{\mathrm{accel}}$ means accelerator capacity cost assigned to the workload. The effective price $r_{\mathrm{eff}}$ is the price per billed accelerator-hour after commitments, discounts, and allocated ownership costs. $H_{\mathrm{billed}}$ denotes aggregate accelerator-hours paid for, including idle reservation; $N_{\mathrm{accept}}$ counts the accepted results produced by those hours. Their ratio gives $u_{\mathrm{accel}}$, the accelerator capacity cost per accepted result.

The ratio $N_{\mathrm{accept}}/H_{\mathrm{billed}}$ is realized throughput. It already includes idle reservation, batching, failed work, and the observed request mix. Do not divide by utilization again. If instead a laboratory benchmark supplies full-capacity throughput, then utilization can be used once to estimate realized throughput, but that estimate must be checked against production telemetry.

Token-based reporting is useful only with the token categories and request mix attached. Providers may price input tokens, cached tokens, and output tokens differently. Self-hosted prefill and decode stress hardware differently. Tool calls, speculative work, rejected outputs, and internal reasoning may add work that is absent from the visible response. Report prompt and output-length distributions, cache-hit rate, batch and concurrency policy, p50, p95, and p99 latency, errors, and accepted-result throughput on the same line as unit cost.

The serving chapters in Part V explain the technical levers. Batching in Chapter 32 can raise realized throughput; decoding and kernel work in Chapter 33 and Chapter 34 can reduce billed hours for the same accepted workload. These Part V serving levers improve economics only when the quality and service gates still pass. Evaluate serving optimizations in Part V against the complete ledger, because engineering time, larger replicas, or a lower cache-hit rate can offset an isolated kernel gain.

Compute is sold with different promises

An hourly quote does not identify the product being bought. Procurement options carry different rights and risks. Commitment risk belongs in the comparison even when the quoted hourly rate is lower:

AWS documentation, for example, distinguishes on-demand, spot, savings commitments, capacity reservations, and scheduled accelerator capacity; spot instances can be interrupted with a two-minute notice (Amazon Web Services 2026). That product detail should not be generalized to every provider. The method is general: convert the quote into effective cost after unused commitments, interruptions, checkpoint and restart cost, regional constraints, storage, network, and egress.

Power and cooling are not external to the ledger. The IEA's 2025 analysis separates servers, storage and networking, and facility infrastructure when it models data-center electricity demand (International Energy Agency 2025). Chapter 68 adds the site and time-to-power contract. A cheap accelerator reservation that cannot obtain deliverable power by the required date is not feasible capacity.

Read historical cost claims at their boundary

The economic question changed as models moved from experiments to widely used services. Hoffmann and colleagues' 2022 Chinchilla study estimated the model size and training-token allocation that minimized loss under a fixed training compute budget (Hoffmann et al. 2022). Sardana and colleagues extended that question in 2024 by adding forecast inference demand at a fixed quality target; within their fitted regime, sufficiently high demand favored smaller models trained on more tokens (Sardana et al. 2024). That result is conditional on the quality target, demand forecast, fitted scaling law, and cost assumptions. It does not prove that one model shape minimizes every deployment's lifetime cost.

Cottier and colleagues estimate the amortized accelerator hardware and energy cost of the final training run for roughly 45 frontier models. Their fitted trend is about 2.4 times per year from 2016, with substantial uncertainty (Cottier et al. 2024). It is an estimate for selected frontier models, not audited cash spend and not the whole cost of model development. Research, ablation, data work, failed runs, evaluation, labor, and the acquisition cost of a fleet require separate treatment.

DeepSeek-V3 provides a different kind of record. Its technical report self-reported 2.788 million H800 GPU-hours for the full official training pipeline of a 671-billion-parameter mixture-of-experts model with 37 billion parameters active per token (DeepSeek-AI 2024). The report's dollar proxy applies an assumed hourly rate. Prior architecture, algorithm, and data research and ablation were not included. The GPU-hour figure is reproducible as the authors' stated resource account; it is not an audited total development cost or evidence that another team can obtain the same outcome at the same price.

API price trends require the same discipline. An Epoch AI analysis compared the listed price of reaching a fixed capability threshold on six benchmarks and found sharply different declines by task over its observation window (Cottier et al. 2025). That is stronger than comparing model names, but it remains task-dependent and uses a stated weighting of input and output prices. Listed price is not provider production cost, and a benchmark threshold is not an application's accepted result. Record capability drift and do not extrapolate a short historical decline as a guaranteed forecast.

Compare buy and self-host scenarios

Use discounted, scenario-specific totals before using a break-even shortcut. For one forecast scenario,

$$C_{\mathrm{buy}} = \sum_{t=1}^{T} d_t B_t, \qquad C_{\mathrm{self}} = F_0 + \sum_{t=1}^{T} d_t S_t, \qquad d_t = (1+r)^{-t}.$$

In words, these formulas sum each alternative's interval costs after the discount factor $d_t$ translates them to the same decision date.

Across uncertain scenarios,

$$\mathbb{E}[C_a] = \sum_{s=1}^{S} \pi_s C_{a,s}.$$

Each symbol separates time from uncertainty:

• $C_{\mathrm{buy}}$ and $C_{\mathrm{self}}$ mean the discounted all-in costs of the managed and self-hosted alternatives. The index $t$ denotes one interval in the $T$-interval planning horizon.
• $d_t$ is the discount factor for cash in interval $t$, and $r$ is the decimal discount rate per interval.
• $B_t$ is the buy-side ledger in interval $t$, including input, cached, and output prices, requests, tools, review, network, and support.
• $F_0$ is the initial self-host cost for integration, evaluation, migration, reserved capacity, and launch. $S_t$ is the later self-host ledger for capacity, people, data, network, failure, and exit preparation.
• The index $a$ denotes one candidate alternative. The index $s$ denotes one of $S$ demand, price, capacity, and failure scenarios; $\pi_s$ is its probability weight, with all weights summing to one.
• $C_{a,s}$ means the discounted cost of alternative $a$ under scenario $s$. The expectation $\mathbb{E}[C_a]$ is their probability-weighted mean.

Use at least low, base, and high demand; slower and faster API-price decline; one capacity shortage; one model migration; and one serious incident. Show undiscounted cash by year as well as discounted totals. Expected cost does not replace tail-risk limits: a cheaper mean can still hide an unacceptable outage or cash commitment.

The familiar linear shortcut is

$$V^* = \frac{F}{p-c}, \qquad p > c.$$

Here, $V^*$ means the volume at which the two linear cost lines are equal. $F$ is self-host fixed cost absent from the API alternative, $p$ is constant all-in API price per comparable unit, and $c$ is constant self-host marginal cost per unit. The condition $p>c$ is required for a positive, finite crossing point.

This expression is a diagnostic, not the decision. It assumes the same capability and service, constant prices, linear marginal cost, no capacity steps, no uncertainty, and no switching cost or exit cost. Real self-hosting adds replicas in discrete blocks, while negotiated API pricing may add volume tiers and minimum commitments. Use the interactive figure to understand the geometry, then return to scenario and sensitivity analysis.

The small program makes four assumptions visible. Demand is measured in millions of comparable units; changing only API price, demand, or the fixed capacity needed for a peak can reverse the result.

self_rate = 0.25  # dollars per million comparable units

scenarios = [
    ("base", 1_600_000, 2.00, 2_650_000),
    ("api-price-down", 1_600_000, 1.30, 2_650_000),
    ("demand-down", 800_000, 2.00, 2_650_000),
    ("peak-capacity-up", 1_600_000, 2.00, 3_250_000),
]

for label, volume, api_rate, self_fixed in scenarios:
    buy = volume * api_rate
    self_host = self_fixed + volume * self_rate
    choice = "self-host" if self_host < buy else "API"
    print(
        f"{label}: buy=${buy/1e6:.2f}M "
        f"self=${self_host/1e6:.2f}M -> {choice}"
    )

Model demand as a distribution

Average demand is not a capacity plan. Record the arrival distribution, request and response sizes, concurrency, geography, and bursts. A self-hosted fleet needs headroom for maintenance, failures, and the chosen tail-latency objective. Once a peak crosses the next replica or cluster boundary, cost rises as a capacity step rather than a smooth marginal line.

Evaluate at least a representative steady interval, a p50 day, a p95 peak, and a p99 stress interval. Replay a burst large enough to form a queue, remove one failure domain, and measure recovery. For training and batch work, record deadline flexibility and checkpoint cadence; those properties determine whether spot interruption saves money after wasted work and restart cost.

Demand uncertainty also affects the buy side. Quotas can prevent nominally elastic capacity from arriving, volume discounts can make the price nonlinear, and a provider revision can change quality or token use. Lock quotes and model revisions to dates, then rerun the comparison when either changes.

Keep cash, accounting, and economic cost separate

capital expense (CapEx) is cash committed to long-lived assets such as owned accelerators and data-center equipment. API calls, rented compute, electricity, and many operations costs are usually operating expense (OpEx), recurring operating expenses. Model development itself is not automatically one or the other: cloud training and research labor can be operating expense, while owned training hardware is a capital asset. Retraining and evaluation also recur.

Three views answer different questions:

• Cash flow records when money enters or leaves the organization. It tests liquidity and financing needs.
• Accounting expense follows the applicable reporting policy. Depreciation allocates the recorded cost of an asset across an estimated useful life.
• Economic cost includes the best forgone alternative, or opportunity cost, plus operational risk, switching cost, and expected salvage value.

Changing a depreciation estimate changes the timing of accounting expense; it does not change the cash already paid. Amazon, for example, reported that it extended server lives from five to six years in 2024, then shortened a subset of servers and networking equipment back to five years in 2025 (Amazon.com, Inc. 2026). That filing demonstrates estimate revision, not the correct life for every accelerator. A team comparing owned with rented capacity should show useful-life and salvage-value sensitivity, physical failures, performance obsolescence, and the planned secondary use of older hardware.

Financing is a fourth view. Debt, leases, or special-purpose vehicles change cash timing, required returns, covenants, and who bears residual risk. They do not make the underlying capacity free. BIS researchers documented rising bond issuance and off-balance-sheet structures in the 2025 infrastructure buildout (Eren et al. 2026). For a product decision, financing belongs in the discount rate, fixed commitments, and downside scenarios. It should not be mixed into a token-cost numerator without an explicit allocation rule.

Operate the economic model

The decision process is a control loop rather than a one-time spreadsheet:

1. Freeze the workload. Version the model candidates, request mix, acceptance rule, service objectives, policy boundary, horizon, and demand scenarios.
2. Screen feasibility. Remove alternatives that fail quality, latency, availability, privacy, security, license, region, or delivery-date gates.
3. Measure the candidates. Benchmark the exact API revision and the exact self-host tuple under the same workload, including failure and peak tests.
4. Normalize the ledger. Apply the same cost categories, shared-cost allocation, currency date, and accepted-result denominator.
5. Model scenarios. Vary demand, prices, capacity steps, useful life, incidents, migration, and retirement; publish sensitivity analysis.
6. Exercise the exit. Export state, replace the provider or runtime, restore service, and price the observed switching cost and exit cost.
7. Approve the commitment. Record the chosen scenario, owner, budget, reservation or contract terms, review date, and rollback trigger.
8. Reconcile actuals. Join invoices, telemetry, evaluation outcomes, and incidents each interval; explain forecast error and rerun the decision.

The evidence record should retain quote and invoice versions, model and runtime digests, workload and acceptance-rule digests, billed and useful capacity, token categories, accepted outcomes, people allocations, incidents, and the forecast used for approval. A variance without those inputs cannot distinguish price drift, demand error, quality failure, or an operational regression.

Chapter 77 now widens the view from one buyer's ledger to the value chain that sets supply, bargaining power, and margins. The bridge is precise: this chapter measures what one feasible alternative costs; the next asks why those prices and choices exist. A cheap token is not necessarily a cheap result.

Further reading