Evaluation and Observability

A model that tops a public leaderboard can still misbehave in your product. The benchmark measured a generic capability on a fixed dataset. Your app runs a specific prompt over your retrieval context, inside an agent loop, against live user input that drifts week to week. The discipline that closes this gap has two halves, and by mid-2026 they have largely fused into one tooling category. The practical question is how to choose and wire that tooling without confusing public benchmark quality with product behavior.

Where it sits in the stack: Part VII defines the measurement theory. Chapter 47 measures models in the abstract, Chapter 48 decides whether a gap is real, Chapter 49 defines the human label, Chapter 50 covers model judges, Chapter 51 covers groundedness, Chapter 52 covers multi-step trajectories, and Chapter 53 turns all of that into release gates. Here the object is your running system. It consumes the serving layer (Chapter 82) and the model gateway (Chapter 88), and it produces the feedback signal that feeds back into agents (Chapter 85) and fine-tuning (Chapter 84): eval datasets become preference data, and agent traces are where failures actually surface.

The two halves of the loop

Evaluation asks "is the output good?" against criteria you define. It runs offline, on a curated dataset, in CI, before you ship, and online, sampled from production traffic and scored continuously. The unit of work is a score. Scores come in three flavors that you mix:

• Deterministic assertions: regex, JSON-schema validation, exact match, a latency or cost budget. Cheap, fast, and the first line of defense.
• Reference-based metrics: BLEU and ROUGE and their retrieval cousins like context precision, context recall, and faithfulness.
• LLM-as-judge: a second model grades the first against a rubric. This is the only practical way to score open-ended quality at scale, and the bulk of Chapter 50 is about doing it without fooling yourself.

observability asks "what actually happened?" It captures a trace: the full tree of a request as it moves through your system, every LLM call, retrieval, tool invocation, and nested agent step, with inputs, outputs, timing, token counts, and cost attached to each span. Without it you are debugging a non-deterministic distributed system by reading flat logs.

The two halves connect because you evaluate the traces you observe. A production trace is a ready-made test case. A judge run over sampled spans yields online scores, and a regression first caught on a dashboard gets promoted into the offline suite as a fixture. This loop, observe then score then curate then re-test, is the practical core of "continuously evaluate an LLM app", and it is the figure to keep in mind for the rest of the chapter (Figure 87.2).

The 2026 shakeout

Two acquisitions reshaped the landscape in early 2026, and they tell you the direction of travel: the standalone eval/observability tool is becoming a feature of a larger data or model platform.

• OpenAI acquired Promptfoo (announced March 9, 2026) and is deprecating its own hosted Evals product (read-only October 31, 2026, shutdown November 30, 2026), steering users toward Promptfoo and Datasets. The open-source openai/evals repo persists; the hosted dashboard does not.
• ClickHouse acquired Langfuse (January 16, 2026, alongside ClickHouse's $400M Series D). Langfuse states its OSS and self-host commitment continues.

Ownership is the wrong unit of analysis. The churn is real, so favor tools whose data and instrumentation you can carry elsewhere. That is the case for OpenTelemetry below.

Key players

All license, pricing, and acquisition claims below are as of mid-2026 and attributed in Further reading. Pricing is the published list shape at time of writing and changes often, so treat exact dollars as indicative. Judge accuracy and "best for" claims are workload-dependent and not independently benchmarked here.

Eval harnesses and frameworks

Tracing and observability platforms

The substrate

As of mid-2026 the OTel GenAI spec is still largely experimental ("Development" stability), though core attributes (gen_ai.system, gen_ai.request.model, token counts) are stabilizing. You opt into the latest with OTEL_SEMCONV_STABILITY_OPT_IN. OpenInference runs ahead of the spec and emits to any OTel backend (Datadog, Grafana, Jaeger), which is why it is the practical choice today.

How to choose

The first cut is capability eval versus application eval, because they are different jobs done by different tools.

Pick a capability harness when you are measuring a model: comparing checkpoints, validating a fine-tune (Chapter 84), or reproducing a leaderboard number. Use Inspect when the task is agentic, tool-using, or safety-flavored, and lm-evaluation-harness for classic academic benchmarks.

Pick an application eval tool when you are measuring your system: this prompt, this RAG pipeline, this agent, against criteria that matter to your product. Within application eval, four axes decide the tool:

LLM-as-judge: cost versus trust

A judge is roughly 500x to 5000x cheaper than human review and can approach human-to-human agreement, but only when calibrated. An uncalibrated judge will score confidently and be wrong just as confidently, and you will not see it. As of mid-2026 the consensus practice, covered in depth in Chapter 50, is:

• Validate against a human-labeled sample. Report Cohen's kappa (agreement corrected for what chance alone would produce) and target around $0.8$ before you trust the judge to gate anything.
• Control for position bias. Score both A/B and B/A orderings and count only consistent wins.
• Control for verbosity and self-preference bias. Never use the same model family as both generator and judge.
• Use explicit, criterion-separated rubrics (G-Eval style). Split hard rules into a DAG of checks rather than one fuzzy prompt.

Change the human base rate and watch raw agreement stay high while Cohen's kappa collapses, which is why "the judge agrees 90% of the time" proves almost nothing.

import numpy as np

# A lazy judge that says "good" almost always, scored against human labels.
# Raw agreement looks great, but Cohen's kappa corrects for chance and collapses.
def kappa(m):
    m = np.array(m, float); n = m.sum()
    po = np.trace(m) / n                       # raw agreement
    pe = (m.sum(0) * m.sum(1)).sum() / n**2    # agreement expected by chance
    return po, (po - pe) / (1 - pe)

n = 1000
for good_rate in [0.70, 0.80, 0.90, 0.95]:
    good = int(n * good_rate); bad = n - good
    caught = int(bad * 0.20)                    # judge flags only 20% of bad answers
    miss = int(good * 0.02)                     # and wrongly flags 2% of good ones
    cm = [[caught, miss], [bad - caught, good - miss]]
    po, k = kappa(cm)
    verdict = "trust" if k >= 0.8 else "do NOT trust"
    print(f"human good={good_rate:.0%}  agreement={po:.2f}  kappa={k:.2f}  -> {verdict}")

A sensible default

For a typical team shipping one or two LLM features and wanting a continuous eval loop without standing up a platform team, here is a stack that is OSS-first, self-hostable, OTel-native, and CI-gated, with a clean upgrade path to a managed platform later.

Two caveats to this default. First, the churn: Promptfoo is now OpenAI-owned and Langfuse is ClickHouse-owned, so re-confirm roadmap and licensing before you commit long-term. Second, do not add a vendor you do not need. Teams already standardized on Databricks should prefer MLflow 3 Evaluate, and teams on Weights & Biases should prefer Weave, over bolting on a new platform.

Wiring

The loop closes through the model gateway (Chapter 88): the app under test, the evals, and the judge all call it, never providers directly.

Here is a minimal Promptfoo config that points at the gateway and uses an LLM judge with a deterministic guard in front of it. Save it as promptfooconfig.yaml and run npx promptfoo@latest eval.

# promptfooconfig.yaml
providers:
  - id: openai:chat:gpt-5            # resolved by the gateway, not OpenAI directly
    config:
      apiBaseUrl: https://gateway.internal/v1
      apiKey: ${GATEWAY_VKEY}        # scoped virtual key, budget-capped
prompts:
  - "Answer using only the context:\n{{context}}\n\nQ: {{question}}"
tests:
  - vars: { question: "...", context: "..." }
    assert:
      - type: is-json                # deterministic guard, runs first and free
      - type: llm-rubric             # LLM-as-judge, calibrated rubric
        value: "Answer is faithful to the context and cites it."
        provider: openai:chat:gpt-5-mini   # judge via gateway, different family

To gate merges, run that one command in CI and fail the job on a regression:

# CI step: fail the build if any eval assertion regresses
npx promptfoo@latest eval -c promptfooconfig.yaml --no-cache --output results.json
# exit code is non-zero when assertions fail, which fails the pipeline

The heartbeat of the whole system is the promotion step. A production trace from Langfuse or Phoenix that fails an online judge gets promoted into the tests: set above, so the offline suite hardens against the failures you actually see in production rather than the ones you imagined at design time. Without that promotion step you have two disconnected tools. With it you have a loop.

The production evaluation loop

That loop earns its keep only when it changes all three operating axes. Observability makes the system more capable by turning nondeterministic behavior into something you can debug, and evaluation turns "it feels better" into a release gate. The gateway keeps online judging cheap enough to run continuously, while deterministic assertions catch the cheap failures before a judge call is made. Trust rests on calibration and portability: an LLM judge is only as trustworthy as its agreement with humans, and the OTel substrate is what lets you carry traces and scores away from any vendor that changes its terms. The loop is worth building only if all three hold, and in mid-2026 the tooling has finally matured enough that they can.

Further reading