Choosing a Model

A model choice is a production decision, not a leaderboard lookup. The thing being chosen is a versioned served system: a particular model revision, provider or serving build, prompt, tools, settings, safety controls, and fallback behavior. A different prompt template or tool loop can change the result even when the weights stay fixed.

Start with a workload contract. It states which tasks matter, what counts as an accepted result, which hard constraints apply, and the limits for quality, cost, latency, availability, and operational risk. Then compare eligible candidates under the same contract. There is no universal winner; there is only a defensible choice for a declared workload and time period.

This chapter turns that idea into an operating procedure. The closed-to-open axis is one input, not the decision procedure: Chapter 73 explains how to inspect a release, while Chapter 47 explains what a public score can and cannot establish.

Choose the system, not the name

A family name such as “Model Pro” is too vague to reproduce. Record the exact candidate identity before testing it:

• provider and model ID, or artifact digest for downloaded weights;
• endpoint and region;
• prompt template and message formatting;
• tool schemas and tool-loop implementation;
• decoding and reasoning settings, including budgets and stopping rules;
• safety policy and content filters;
• cache, batch, router, and fallback policy; and
• rate and capacity limits used during the test.

Call this configuration $m$. If any behavior-changing field changes, treat the result as a new candidate. A dated record matters even when an API exposes a fixed-looking name: service implementations, quotas, and terms can change outside the repository.

The emphasis on a whole system is not new. Liang and colleagues introduced HELM in 2022 and published it in 2023 to compare language models across scenarios and multiple metrics under standardized conditions (Liang et al. 2023). Mitchell and colleagues introduced model cards in 2019 so that intended uses, evaluation conditions, and limitations travel with a model release (Mitchell et al. 2019). The practical extension is simple: keep a card for the served configuration that your application actually calls.

The order in Figure 81.1 prevents an attractive score from hiding a legal, interface, or capacity failure. It also prevents a cheap but inadequate system from entering an economic comparison.

Gate eligibility before ranking

Write hard requirements as predicates. For candidate $m$, one compact gate is

$$E(m)=H_{\mathrm{license}}(m) \land H_{\mathrm{data}}(m) \land H_{\mathrm{region}}(m) \land H_{\mathrm{interface}}(m) \land H_{\mathrm{capacity}}(m).$$

Here $H_{\mathrm{license}}(m)$ means the legal review permits the intended use; $H_{\mathrm{data}}(m)$ means service handling and retention satisfy policy; $H_{\mathrm{region}}(m)$ means deployment and processing locations are allowed; $H_{\mathrm{interface}}(m)$ means required modalities, tools, and output forms work; and $H_{\mathrm{capacity}}(m)$ means the candidate can meet the required load and availability. Every factor is Boolean. Unknown does not pass. An unresolved item goes through an explicit exception process rather than becoming a low penalty in a weighted score. Eligibility is not a weighted score.

The gate should cover more than these five examples when the workload requires it: accessibility, safety controls, export restrictions, audit evidence, indemnity, data deletion, or an exit path can all be hard requirements.

“Open” is not enough information

Keep the artifacts and the permissions in separate columns. A release may provide weights without training data, training code, or a useful model card. A permissive code license attached to one repository does not automatically describe the terms for weights or data. A hosted service adds another contract covering data retention, training on customer data, regions, acceptable use, rate limits, deprecation, and support.

The Open Source Initiative's 2024 definition asks whether a system grants the freedoms to use, study, modify, and share, and whether the preferred form for modification includes the needed data information, code, and parameters (Open Source Initiative 2024). That is a much richer claim than “the weights can be downloaded.” Do not infer rights from the word open. Inspect each artifact, license, and service agreement that applies to the exact deployment. This is an engineering review framework, not legal advice.

Hosted, managed, and self-hosted are deployment choices

Artifact access and deployment are related but not identical. Downloadable weights can run on owned machines, rented accelerators, or a managed endpoint. A hosted API can be reached directly or through a cloud marketplace. Its service terms are distinct from an artifact license. Compare the actual alternatives available to the organization, including their capacity, staffing, security, and exit costs.

Use public evidence to discover candidates

Public benchmarks are useful for candidate discovery. They are weak grounds for a production decision unless their task, system, and measurement process match yours. HELM's scenario-and-metric framing is valuable precisely because it makes those choices visible (Liang et al. 2023).

Ask four questions about every published result:

1. Construct match: does the benchmark measure the capability the product needs, or a convenient proxy?
2. Harness match: are the prompt, tools, sampling, context, budget, and grader comparable to production?
3. Uncertainty: are item counts, repeated runs, intervals, and close ranks reported well enough to support the claimed difference?
4. Data provenance: when and how were tasks created, exposed, filtered, and scored?

Also identify the system being measured. An agent benchmark usually measures a model together with a scaffold, tools, retries, and budget. SWE-bench, for example, evaluates issue resolution in real repositories, not isolated code completion (Jimenez et al. 2024). LiveCodeBench continuously draws newer competition problems and covers several coding behaviors (Jain et al. 2024). These design choices address different questions; neither score transfers automatically to a private workload.

Human-preference boards answer yet another question. Chatbot Arena uses paired user preferences and a statistical ranking method (Chiang et al. 2024). That is evidence about aggregate preference under its traffic and interface, not direct proof of factual correctness, policy compliance, or tool reliability.

A suspicious public-to-private gap justifies investigation. It cannot prove contamination by itself: task mix, prompts, tools, budgets, graders, and chance can also create the gap. Record the possible explanations and run controlled tests instead of assigning a cause from one comparison.

Build the evaluation around the workload

The internal evaluation is the decision instrument. Freeze its contract before looking at final results:

• define workload strata, such as language, task type, difficulty, risk, and input size, with production weights;
• write the acceptance rule and primary quality measure;
• name the incumbent baseline and the smallest practically important change;
• separate tuning cases from a locked confirmation set;
• declare cost, latency, critical-failure, and availability limits; and
• choose a precision target and sample size plan.

Run every candidate on the same cases. Randomize execution across time when provider load may drift, and use repeated trials at production settings when generation is stochastic. Preserve the independent unit: several turns from one conversation or several attempts on one issue are clustered observations, not unrelated evidence.

For a challenger $m$ and baseline $b$, define the paired result on independent case $i$ as

$$d_i = y_i^{(m)}-y_i^{(b)},$$

where $y_i^{(m)}$ and $y_i^{(b)}$ are scores produced under the frozen acceptance rule. Before running the comparison, choose a non-inferiority margin $\delta>0$: the largest quality loss the team would accept for gains elsewhere. The challenger clears the quality gate only when the lower confidence interval for the workload-weighted mean difference is greater than $-\delta$. Require a lower bound above zero if the claim is superiority.

This is a decision rule, not a universal statistical recipe. Binary paired outcomes, rubric scores, clustered conversations, and repeated stochastic runs need different interval methods; Chapter 48 develops those choices. Pick sample size from the precision target, expected paired variance, cluster structure, and desired error rates. There is no defensible magic count. Inspect slice results and critical failures even when the aggregate passes.

Prefer executable or rule-based graders where they represent acceptance. When human judgment is necessary, blind the reviewer to candidate identity and randomize presentation order. When an LLM judge is necessary, pin its full configuration, validate it against human labels, measure judge disagreement, swap answer order, and adjudicate important conflicts. LLM judges can exhibit position bias (Wang et al. 2024); Chapter 50 and Chapter 52 cover the broader evaluation design.

A reviewable configuration might look like this:

# eval-contract.yaml: illustrative fields, not universal thresholds
workload: evals/support-confirmation-v8.jsonl
strata: [language, request_type, risk, input_size]
candidate_systems:
  - config: systems/incumbent-2026-08-01.yaml
  - config: systems/challenger-a-2026-08-01.yaml
primary_metric: accepted_result
comparison:
  design: paired
  repeats_per_case: 3
  noninferiority_margin: declared-before-run
  confidence_method: chosen-for-independent-unit
secondary_limits: [critical_failure_rate, cost_per_accepted_task, p95_latency]
release: [shadow, canary, rollback]

Compare total cost for accepted work

Token price is an input, not the economic result. Tokenizers, cache rules, reasoning tokens, output length, tool use, retries, and acceptance rates differ. Measure end-to-end cost per accepted task over the same accounting horizon and workload.

Here $s$ identifies a workload stratum and $r$ a repeated trial. Let $w_s$ be the production weight of stratum $s$, $K_{msr}$ the realized end-to-end cost of candidate $m$, and $A_{msr}$ equal one when the result is accepted and zero otherwise. Then

$$C_m= \frac{\sum_s w_s\sum_r K_{msr}} {\sum_s w_s\sum_r A_{msr}}.$$

$C_m$ is undefined when the weighted accepted count is zero, which is the right warning. In this formula, the ledger behind $K_{msr}$ should include input tokens, output tokens, cache writes and reads, tool calls, retries, network charges, human review, and expected incident cost. For self-managed serving it also includes accelerators, CPU and memory, storage, idle capacity, operations labor, on-call work, and software. Keep cost per attempt beside the accepted-task measure so a change in the denominator remains visible.

Measure client-side latency as well. Report first-token and end-to-end p50, p95, and p99 by workload stratum and offered load. Separate queueing, prefill, decoding, tools, and retries when diagnosing the result. A routing cascade can reduce average spend yet worsen the tail because escalated requests pay for multiple stages.

Hosted versus self-hosted crossover

For a quick sensitivity analysis, suppose two quality-equivalent alternatives meet the same service objective. Over one accounting period, write

$$C_h(N)=F_h+Nv_h$$

where the subscript $h$ denotes hosted service, and

$$C_s(N)=F_s+Nv_s$$

for self-hosted service. $N$ is accepted tasks in that period; $F_h$ and $F_s$ are fixed costs; and $v_h$ and $v_s$ are variable costs per accepted task. Use the same currency and the same accounting horizon for every term. When $v_h>v_s$, the linear crossover is

$$N^*=\frac{F_s-F_h}{v_h-v_s}.$$

A positive crossover exists only when the numerator and denominator imply $N^*>0$. Otherwise there is no positive crossover in this linear range. Real systems add capacity steps, demand bursts, commitments, utilization, migration costs, and changing prices; model low, base, and high scenarios in Chapter 76. The shortcut is invalid when the alternatives are not quality-equivalent or miss different service objectives.

Select a frontier, not a score champion

After the hard gate and quality gate, place the survivors on a Pareto frontier. A candidate is dominated when another eligible candidate is at least as good on every decision-relevant measure and better on one. Remove it. The remaining trade-off may still require judgment: one team may pay more for a lower critical-failure bound, while another may accept slower responses for data residency.

Do not hide that judgment inside arbitrary normalized weights. Record the trade-off, owner, evidence, and expiry date. Research on learned cascades shows that routing can improve a measured cost-quality trade-off (Chen et al. 2023; Ong et al. 2024), but a router is itself a versioned system. Evaluate the policy end to end, including misroutes, verifier errors, duplicate work, fallback latency, and side effects.

Wire the decision so it stays true

Put provider-specific behavior behind a provider adapter when the application needs portability or centralized controls. A gateway, a routing and policy layer for model calls, is one design option for shared keys, budgets, observability, and routing. It also adds a dependency and cannot make providers semantically identical.

An OpenAI-compatible transport usually covers only a common subset. Tool semantics, structured output, tokenization, caching, streaming, multimodal inputs, reasoning controls, and error behavior can still differ. A policy-compatible fallback must satisfy the same data and regional rules, and its behavior must be validated by the same evaluation contract. Test ambiguous timeouts and action idempotency before enabling automatic failover. See Chapter 88 and Chapter 90.

Pin stable revisions where the provider or artifact supports them. Otherwise record the exact endpoint, region, configuration, and observation date, then run sentinel evaluations frequently. Store selected-model and route decisions with each trace so failures can be attributed to the system that handled them.

Promote through shadow traffic, then a small canary with automatic rollback. Monitor acceptance, critical failures, latency tails, spend, capacity, and provider errors by stratum. Define every re-evaluation trigger in advance: model or prompt change, price or terms change, new tool schema, workload drift, quality drift, repeated capacity failures, or the decision record's expiry date.

The output is a decision record: the workload contract, candidate identities, eligibility evidence, evaluation results with uncertainty, cost ledger, Pareto trade-off, rollout owner, rollback rule, and re-evaluation trigger. That record makes a model choice explainable today and replaceable tomorrow.

Further reading

Primary and archival sources for the framework and its limits.