Evaluating Agents and Capabilities

An agent does not merely produce an answer. It observes an environment, chooses actions, calls tools, and changes state. Evaluating one therefore requires more than reading its final message. The evaluation must run the complete system in a controlled environment and check what actually happened.

This distinction is easy to miss. An agent may say that it issued a refund while the customer database remains unchanged. It may produce a working patch after reading a secret test file. It may reach the correct page while violating an identity check that the policy required first. The final sentence is evidence about what the agent claims. The environment state and retained action log are evidence about what it did.

A reliable evaluation follows one rule:

Evaluate the versioned system, verify the resulting state, and inspect the trajectory only where the task imposes genuine process constraints.

From episodic return to tool-using agents

The underlying measurement problem predates language models. Reinforcement learning formalized an agent as a policy interacting with an environment over an episode, with performance measured from the rewards produced by that interaction (Sutton and Barto 2018). Modern language-model agents preserve the episode but replace a compact simulator with browsers, repositories, databases, APIs, and human conversations. The difficult part is no longer merely collecting a numeric reward. It is specifying which real-world state counts as success, recreating the starting state, and distinguishing an agent error from a broken tool or evaluator.

Agent benchmarks made that shift explicit. AgentBench combined several interactive environments (Liu et al. 2024). WebArena used self-hosted websites and functional task checks (Zhou et al. 2024). SWE-bench asked systems to edit a repository so that an issue-specific test harness passes (Jimenez et al. 2024). OSWorld configured real desktop states and evaluated the state left after GUI actions (Xie et al. 2024). GAIA instead used questions whose short answers may require browsing, files, code, or multimodal reasoning (Mialon et al. 2024). These benchmarks do not measure one interchangeable quantity. They illustrate several ways to turn an interaction into checkable evidence.

Define the system under test

An agent score belongs to a configuration, not to model weights alone. At minimum, the configuration includes:

• the model revision, serving route, decoding settings, and reasoning budget;
• the system prompt, tool descriptions, orchestration loop, and context policy;
• the tool implementations, credentials, permissions, and network policy;
• the sandbox image, dependency versions, initial data, and reset procedure;
• the task, simulated user if any, time or turn budget, and grader revision.

The first two groups are the model and Chapter 41. The others determine what the system can observe and change. A new prompt, browser controller, context compactor, or timeout can move the score without any change to the model. Report such a result as a system result. A model comparison is justified only when the remaining configuration is held fixed, and even then the fixed harness may favor one model's API conventions over another's. Controlled model-harness pairings in Harness-Bench demonstrate why the configuration, rather than either component in isolation, is the defensible unit of reporting (Yao et al. 2026).

A useful task contract makes the condition executable:

AgentTask:
  task_id, task_revision, source, slice_labels
  initial_state_fixture, reset_check
  user_request, available_tools, permissions
  hard_process_constraints
  terminal_conditions, step_limit, time_limit
  outcome_assertions, partial_credit_rubric
  grader_revision, grader_test_manifest

AgentRun:
  task_id, system_spec_hash, attempt, seed
  initial_state_hash, observation_action_log
  final_state_hash, terminal_reason
  outcome_assertion_results, constraint_results
  partial_score, cost, latency
  infrastructure_status, artifact_locations

This record separates the task definition from one stochastic attempt. It also prevents silent changes to the environment or grader from masquerading as model progress.

A simulated user is part of the instrument, not neutral background. Its model, prompt, persona, available information, and stopping behavior can change the conversation and therefore the score. Pin those choices, sample production-like variation, and compare a subset of simulated interactions with representative humans before using the result as evidence about human-agent interaction.

Score state and constraints separately

Let one run on task $i$ produce the trajectory

$$\tau_{ir}=(s_0,o_0,a_1,o_1,\ldots,a_T,s_T).$$

Here $i$ identifies the task; $r$ identifies a repeated attempt; $s_0$ and $s_T$ are the initial and final environment states; $o_t$ is the observation available at step $t$; $a_t$ is the action chosen after that observation; and $T$ is the step at which the run terminates. The exact state may not be visible to the agent. It must still be visible to the evaluator wherever the evaluator claims to verify it.

Define $G_i(s_0,s_T)\in\{0,1\}$ as the task's outcome check. For a refund task, it might verify a new database record, the correct amount, and the expected account balance. Let $C_{ij}(\tau_{ir})\in\{0,1\}$ check hard process constraint $j$, such as obtaining authorization before issuing the refund. If task $i$ has $J_i$ hard constraints, required success is

$$Y_{ir}=G_i(s_0,s_T)\prod_{j=1}^{J_i} C_{ij}(\tau_{ir}).$$

$Y_{ir}$ represents required success and is one only when the outcome and every hard constraint pass. When $J_i=0$, the empty product is one, so success depends only on the outcome. This formulation does not require the agent to follow an author's preferred route. It checks the final state and only those intermediate facts that are part of the real requirement.

The distinction matters:

Outcome checks should normally be independent of the agent's output channel. Read the database through an evaluator credential, run tests from a clean checkout, or query the browser backend. When only a model judge can assess the artifact, apply the validation and confirmation discipline of Chapter 50. A model grading another model is not an independent state check merely because it receives a different prompt.

Make every task valid and resettable

A runnable task is not automatically a valid task. Before using a case, verify four things.

First, the request must be solvable from the information and permissions given. Have a competent human complete it from the same initial state, under the same policy and budget where practical. Second, the outcome checks must accept more than the reference path. Tests should describe required behavior, not reproduce the original developer's patch. Third, known bad solutions must fail. Seed wrong-account refunds, incomplete patches, forbidden actions, and superficially correct final messages. Fourth, the reset must restore every state the next run can observe, including database rows, files, browser sessions, clock-dependent data, rate limits, and caches.

SWE-bench provides a useful warning. Its original tasks paired real issues with repository tests, but professional review later found cases with underspecified issues or inappropriate tests and produced a 500-case Verified subset (OpenAI 2024). A later audit still found material problems among frequently failed Verified cases (OpenAI 2026). Executable grading did not fail as an idea. The audits instead show that a grader is software and needs tests, versioning, and continuing review like any other critical component.

Long tasks often need partial credit, but the rubric must describe observable work. PaperBench decomposed research replication into a hierarchy of gradable requirements and separately evaluated its automated judge (Starace et al. 2025). That pattern is useful beyond research: preserve the binary end-to-end completion rate, add component scores for diagnosis, and validate the component grader. Do not replace the difficult end-to-end result with an easier average of milestones.

Separate agent failures from evaluation failures

Every run needs a terminal reason, not just a zero. A practical taxonomy is:

• success: the outcome and all hard constraints pass;
• agent failure: the agent stops, times out, takes a prohibited action, or leaves an incorrect state while the evaluation machinery remains healthy;
• harness compatibility failure: the chosen model cannot use the harness or tool protocol as configured;
• environment failure: a service, dependency, network route, or reset is broken independently of the agent's decisions;
• grader failure: the run completes but the evaluator crashes, cannot read the state, or produces an indeterminate verdict.

These categories support different decisions. Agent failures count against the tested system. Environment and grader failures usually produce invalid attempts that are rerun after repair, with their rates reported. Compatibility failures belong to the system configuration and should not disappear; they show that the pairing cannot perform the task as deployed.

Predeclare the policy. Otherwise a team can inflate a score by relabeling weak runs as infrastructure problems after reading the trace. Retain service logs and health probes that justify each exclusion. Run a cheap smoke task before an expensive batch, and fail the batch when reset checks or canary assertions fail.

One attempt is not reliability

Agents are stochastic and interactive. The same configuration can take a new path on each attempt, while a simulated user or mutable service adds further variation. Let

$$p_i=\Pr_R(Y_{ir}=1)$$

be task $i$'s probability of required success under the declared run distribution $R$. Here $Y_{ir}$ is the binary result defined above, and $R$ includes the production-like randomness in decoding, users, tools, and environment conditions. Under independent attempts with the same $p_i$,

$$\operatorname{pass@}k_i = 1-(1-p_i)^k.$$

$$\operatorname{pass}^k_i = p_i^k.$$

The first quantity is pass@k, the probability of at least one success in $k$ attempts. It measures whether the system can succeed when retries are allowed. The second is the probability that all $k$ attempts succeed. It measures repeated reliability. $k$ is the number of attempts; neither metric creates new tasks. $\tau$-bench introduced pass$^k$ while evaluating tool-using agents in dynamic conversations and checking final database state against an annotated goal (Yao et al. 2025).

Set a per-attempt success probability and inspect how the two questions diverge.

import numpy as np
import matplotlib.pyplot as plt

p = 0.6
k = np.arange(1, 11)
at_least_one = 1 - (1 - p) ** k
every_attempt = p ** k

plt.figure(figsize=(5, 3))
plt.plot(k, at_least_one, "o-", label="pass@k: at least one")
plt.plot(k, every_attempt, "s-", label="pass^k: every attempt")
plt.xlabel("attempts, k")
plt.ylabel("probability")
plt.ylim(0, 1)
plt.legend()
plt.tight_layout()
plt.show()

The closed forms assume independent, identically distributed attempts and known $p_i$. Real evaluations estimate $p_i$ from few repetitions, and failures may be correlated through shared services or user simulators. With $n$ valid attempts and $c_i$ observed successes for task $i$, the first finite-sample formula below estimates at least one success, while the second estimates $k$ successes. These are the task estimators used by $\tau$-bench.

$$\widehat{\operatorname{pass@}k}_i =1-\frac{\binom{n-c_i}{k}}{\binom{n}{k}}.$$

$$\widehat{\operatorname{pass}^k}_i =\frac{\binom{c_i}{k}}{\binom{n}{k}}, \quad k\le n.$$

Here $\binom{a}{k}$ counts the subsets of $k$ attempts chosen from $a$ attempts. Average these task-level estimates with the suite's declared task weights. Keep repetitions nested inside tasks, report the number of valid attempts, and use the paired and cluster-aware methods of Chapter 48. Do not compute pass$^k$ by raising a pooled suite accuracy to the $k$th power when tasks have different success probabilities.

Reliability also changes with task horizon. More steps create more places for tool, planning, context, and recovery failures. A suite should therefore report success by meaningful length or difficulty slices rather than allowing many short tasks to hide collapse on longer work. Length is not a causal explanation, however: tool quality, observability, and task composition can change with it.

Use traces for diagnosis, not retrospective storytelling

Once outcomes are scored, traces can locate where the system failed. Preserve observations, tool requests and responses, state-changing events, model usage, terminal reason, and grader evidence. Then apply a fixed failure taxonomy such as perception, planning, tool selection, invalid arguments, policy violation, state verification, or recovery.

Do not assign a root cause from the final outcome alone. A missing refund can come from a planning error, an API rejection, an expired credential, or a grader reading the wrong database. Sample failures for independent adjudication, allow multiple contributing labels, measure agreement on the labels, and turn confirmed evaluator defects into regression tests. The purpose of the trace is to improve the system and its measurement, not to rescue a preferred score with a persuasive post-hoc explanation.

The operating contract

A release-grade agent evaluation can be summarized as nine commitments:

1. Name the deployment decision and target task population.
2. Version the model, harness, tools, permissions, environment, budgets, user simulator, and grader as one system specification.
3. Prove that each task is solvable and that its reset restores a checked initial state.
4. Verify completion from environment state; use the trace only for declared constraints and diagnosis.
5. Test the grader with known successes, known failures, and adversarial near misses.
6. Predeclare terminal reasons, invalid-run handling, retry rules, and partial credit.
7. Repeat stochastic tasks and report both capability and reliability metrics with their denominators and uncertainty.
8. Retain per-run artifacts so every aggregate can be reconstructed.
9. Keep a locked confirmation suite and carry validated production failures back as regression cases, following Chapter 53.

Further reading