Choosing a Model

The first practical decision in any stack is which model to call. It looks like a quality question and is mostly not. The practical sequence starts by placing a model on the closed-to-open axis, reading its license for what it actually permits, triangulating three leaderboards without trusting any one of them, and running a small eval that decides the call. It ends by pinning and wiring the result so that a vendor's weekly point release does not silently change your product. The companion chapters are Chapter 73, which frames the openness axis, and Chapter 47, which frames why a published score is a claim about a model and its harness together.

The master axis: rent or own

One axis predicts almost everything else about a model choice, and it is not quality. It is whether you rent the capability behind a metered API or own the weights and run them yourself.

A frontier hosted model is the most capable general model a lab is willing to expose, served behind a per-token API. You send tokens, they bill you, you never hold the weights. The lab owns the serving stack: batching, KV-cache, speculative decoding, autoscaling, and the accelerators (Chapter 82). You inherit their latency distribution and their rate limits, and you consume the serving optimizations as knobs (cached-input pricing, batch and priority tiers) rather than operating them.

An open-weight model is one whose trained parameters you can download and run on your own hardware. You own the operational burden and, in exchange, you control cost, latency, data residency, and the right to fine-tune. The model family you pick sets what your inference server (vLLM, SGLang, llama.cpp; see Chapter 82) must load, how much VRAM it needs, and the throughput you can reach.

That is the same trade the book draws throughout: rent capability and give up control of the serving stack, or host weights and own the operations. In 2026 the gap that used to make this a quality compromise has mostly closed. The best open weights now sit within a point or two of closed frontier models on coding and reasoning boards, which turns "self-host versus call an API" into a genuine architecture decision rather than a concession (per Artificial Analysis and open-LLM leaderboards, as of 2026-06).

Licenses outrank benchmarks

For open weights, the license decides more than the benchmark does, because a license can disqualify a model for your product no matter how well it scores. Three tiers matter.

• Truly open (Apache-2.0, MIT). No user caps, no field-of-use limits, no geographic carve-outs. Qwen, DeepSeek, Mistral's current flagship, Gemma 4, gpt-oss, GLM, and OLMo sit here as of mid-2026. Pick these when you might scale past tens of millions of users, operate in the EU, or want zero field-of-use ambiguity.
• Source-available (restricted). Weights are downloadable but the license adds caps or carve-outs. Meta's Llama 4 Community License permits commercial use only below 700M monthly active users and excludes multimodal features for EU-domiciled licensees, with an acceptable-use policy on top. This is not OSI-open. Read the actual LICENSE before you build on it.
• Closed (no weights). The hosted-API frontier. You get a contract and an endpoint, not parameters.

A second, orthogonal distinction is open-weight versus fully-open. Most "open" models release weights only; the training data and full recipe stay private. A smaller set (AI2's OLMo) releases data, code, and checkpoints too. If you need reproducibility, auditability, or the ability to retrain from scratch, that difference is the whole game, and it is independent of the license tier.

The durable rule: for anything that might scale or ship in a regulated region, read the license first and the leaderboard second.

The players, dated and attributed

The two tables below are a mid-2026 snapshot. Pricing is per-1M-token input/output for standard synchronous calls; batch tiers typically run about half price and cached input is heavily discounted.

Frontier hosted models

Open-weight families

Parameter notation is Total / Active for Mixture-of-Experts (Chapter 9); A3B means about 3B active parameters per token.

Leaderboards narrow candidates

A leaderboard is a shortlisting tool, not a verdict. The single most important practitioner lesson from Chapter 47 is that a published score is a claim about a model and its measurement harness together, not about the model alone. Independent re-runs of identical weights swing 10 to 20 percentage points depending on prompt template, system prompt, tool scaffolding, sampling parameters, and grader. The recurring 2026 pattern: a model's score on a public, verifiable coding board runs markedly higher than the same weights on a held-out harness, and that gap is the harness plus any contamination, not a change in the model.

So read three boards, not one, and match the board to the question you are actually asking.

Agreement across the three is a strong signal. Disagreement tells you exactly which axis is fragile. The tell for contamination is a large gap between a model's public-set score and its held-out or post-cutoff score: a small gap is a credibility signal, a large gap is a red flag.

A decision framework

The decision is rarely "which scores highest." It is the intersection of where you are billed, the shape of the task, context-versus-cost, governance, and (for open weights) the hardware you have. Figure 81.2 walks the order in which these actually decide the call.

Stated as rules of thumb, in roughly the order they decide:

• Pick by where you live first. The cloud you are billed through usually wins on procurement, data residency, and committed-use discounts. Heavy AWS shop to Bedrock; Azure to Azure OpenAI; GCP to Vertex AI.
• Pick Claude (Opus 4.8 / Sonnet 5) when the workload is agentic coding, long-running tool use, or safety-sensitive, and you value reliability and pinned, reproducible model IDs.
• Pick GPT-5.5 when you want the widest ecosystem and tooling or you are on Azure.
• Pick Gemini 3.1 Pro Preview when you are on GCP or the input is genuinely multimodal (audio, video, PDF in one model).
• Pick Grok or DeepSeek when price-per-token and very long context dominate and governance allows it.
• Pick an open family (default Qwen3.5/3.6) when you must control cost, latency, residency, or the right to fine-tune (Chapter 84), and you have the hardware. Choose DeepSeek-V4 or GLM-5.2 for agentic coding at the frontier, Gemma 4 for single-GPU, OLMo 3 when reproducibility is a hard requirement.

Long context deserves a separate caution. It is expensive: OpenAI bills prompts at 270K+ at a premium, Gemini roughly doubles input price above 200K. Prefer retrieval over stuffing the window (Chapter 86) when you can; it is usually cheaper and more reliable than relying on raw context size.

Try moving the slider in code: change the threshold and the over-cliff multiplier to see where stuffing the whole window stops paying off versus retrieving a fixed small slice.

import numpy as np
import matplotlib.pyplot as plt
base = 2.0 / 1e6           # $ per input token below the cliff
threshold = 200_000        # tokens; price kink here
mult = 2.0                 # over-cliff tokens cost this much more (marginal)
retrieved = 8_000          # fixed small window when you retrieve instead
n = np.linspace(0, 600_000, 400)
over = np.maximum(n - threshold, 0)
stuff = base * (np.minimum(n, threshold) + over * mult)  # tiered, not flat
retrieve = np.full_like(n, base * retrieved)             # retrieval is flat
plt.plot(n / 1e3, stuff, label="stuff the window")
plt.plot(n / 1e3, retrieve, "--", label=f"retrieve {retrieved // 1000}K")
plt.axvline(threshold / 1e3, color="gray", ls=":", label="price cliff")
plt.xlabel("relevant tokens (thousands)"); plt.ylabel("cost per call ($)")
plt.legend(); plt.title("Long-context cliff vs retrieval"); plt.show()
print(f"At 400K tokens: stuff=${base*(threshold+200_000*mult):.3f}, "
      f"retrieve=${base*retrieved:.3f}")

A sensible default, and why

As of mid-2026, for a typical product team building agent and retrieval-augmented generation (RAG) features and not locked to a specific cloud:

• Hosted default: Anthropic Claude. Sonnet 5 as the everyday workhorse, shipped 2026-06-30 as the successor to Sonnet 4.6 with near-Opus agentic performance at $2 / $10 introductory pricing, escalating to Opus 4.8 for the hardest agentic and coding steps. Both carry pinned model IDs and the family's agentic-coding track record across all four major access layers; a default this new still goes through your own eval gate before it replaces a pinned 4.6. Note the reconciliation from the contested box above: Fable 5 and Mythos 5 topped the boards, vanished under an export directive, and returned three weeks later, so default to a model you can reliably call and keep a pinned fallback. Best on a board is not the same as available to call.
• Strong alternatives: GPT-5.5 for the widest ecosystem or Azure; Gemini 3.1 Pro Preview for GCP or multimodal.
• Self-hosted default: the Qwen3.5/3.6 family (Apache-2.0), the most well-rounded truly-open lineage, with DeepSeek-V4 or GLM-5.2 for frontier agentic coding, Gemma 4 for single-GPU, and OLMo 3 when you need full reproducibility.

Do not hard-code one vendor. Put a gateway, a routing and policy layer for model calls, in front so you can route per task and fail over, and re-evaluate the default every quarter; the gap between the top few is small and the versions churn. Chapter 88 compares the gateway options, including the author's own Lux, named there as a disclosed worked example rather than a neutral pick. This default is a starting posture, not a verdict. The verdict is your own eval.

Pin and eval: the part that actually decides

Public boards narrow the field to three or four candidates. Only your own offline eval picks the winner, because only your eval matches your distribution, your prompts, your tools, and your acceptance bar.

Two disciplines make this real. Pin every model ID in config to a snapshot, never "latest," so a vendor's point release cannot silently change behavior. Treat your eval set as a versioned contract in the repo, like a test suite: a model swap is a diff against it, a prompt change is a diff against it, and CI runs it before any bump. Build it from real traffic (50 to 200 items is enough to start), score with verifiable graders where possible (exact match, unit tests) and an LLM judge only where unavoidable (Chapter 50), and always report cost and latency next to accuracy, never accuracy alone. Logged production failures become next quarter's eval items.

# eval.yaml  (versioned in-repo; CI gate on every model or prompt change)
dataset: ./evals/support_v7.jsonl    # private, from real traffic, versioned
candidates:                          # shortlist from the public boards
  - anthropic/claude-opus-4-8        # pinned snapshot IDs, never "latest"
  - openai/gpt-5.5
  - google/gemini-3.1-pro-preview
graders:
  - type: exact_match                # verifiable wherever possible
  - type: unit_test                  # for code-shaped outputs
  - type: llm_judge                  # sparingly; pin the judge + rubric version
    judge_model: pinned-strong-model
    rubric: ./evals/rubric_v3.md
report:
  axes: [accuracy, cost_per_task, p95_latency]   # never accuracy alone
  ci: bootstrap_95                   # report intervals; a single number lies
gate:
  block_merge_if: accuracy_drop > 2pct OR p95_latency_regression > 15pct

Wiring it into the stack

The app should never talk to a raw vendor SDK. It talks to a gateway that holds keys, enforces per-team budgets and rate limits, normalizes the request shape, and fails over between providers (Chapter 88). For closed APIs there is no serving layer you operate, so the gateway is your control point; for open weights the gateway sits in front of your own inference server, which exposes the same OpenAI-compatible API, so the agent code does not change between them. Figure 81.5 shows the seam.

A provider-agnostic routing config makes the per-task picks and the fallback chain explicit, and guards the long-context price cliff:

# gateway routing (illustrative; every model is a pinned snapshot ID)
default_model: anthropic/claude-sonnet-5
routes:
  - match: { task: hard_coding }      # escalate the gnarly agentic steps
    model: anthropic/claude-opus-4-8
  - match: { task: multimodal }       # audio / video / pdf in one model
    model: google/gemini-3.1-pro-preview
  - match: { task: bulk_cheap }       # high-volume, cost-capped
    model: deepseek/deepseek-v4-flash
fallbacks:
  anthropic/claude-opus-4-8: [openai/gpt-5.5, google/gemini-3.1-pro-preview]
limits:
  long_context_guard: 200000          # warn before the >200K price cliff

The model choice closes on three axes. Capability is what the boards measure and your eval confirms. Set against it is efficiency, the cost-per-task and latency that make any rank meaningless until the price and speed axis sits beside it. Then comes trust: the license, the governance posture, and the availability guarantee, with the three-week Fable 5 outage as the reminder that a model you cannot reliably and lawfully call has no capability for you at all. Pick on all three, pin the result, and let your eval, not a leaderboard, cast the deciding vote.

Further reading

First-hand sources first.