Arc 8: Evaluation & Scientific Discipline

Tool and Agent Evaluation

7 min readbuilds on Tool-Use Fine-Tuning and Evaluating Reasoning

Grade the resulting state

Authored task: normalize a text field; a missing value must become an empty string.

Run A: Edits the function, tests ordinary strings, and stops. The missing-value test still fails.
Run B: Checks the missing-value requirement, patches it, and passes the complete required test suite.
Outcome: Only Run B meets the final-state criterion.
Trace: Run A’s actions help diagnose its failure. Merely calling a test tool is not evidence that its patch works.

An agent can write a convincing completion message while leaving the task unfinished. It can also take an unexpected route and still produce exactly the right result. Evaluating the message or matching a preferred action sequence will miss one of those cases.

Start with the state the task requires: a patch that passes the specified tests, a booking with the correct fields, or a document saved in the right place. Then keep the action trace to explain successes and failures.

This builds on tool-use training and evaluating reasoning. The model, its prompts, tools, environment, and stopping policy together form the system being tested.

Define success before running the agent

A coding task needs an initial repository state, a problem statement, and an evaluator. Specify which tests establish the requested change and which protect existing behavior.

SWE-bench uses real repository issues and evaluates submitted patches with tests. Its Verified subset contains 500 human-filtered tasks selected to improve evaluability. The public issues and code can still be encountered during training; task verification is not a contamination guarantee.

For transactional tasks, inspect the final database state and relevant policy constraints. The original $\tau$ -bench evaluates simulated user interactions with domain tools and policies, including comparison of final database state against the annotated goal. A fluent conversation does not substitute for the required state change.

An evaluator should run even if the agent exhausts its budget, crashes, or never announces completion. Record those stopping conditions separately. Otherwise the reported denominator can quietly exclude the difficult runs.

Use the trace to locate failures

A trace with an actual defect

Authored example. The expected behavior for None is the empty string.

Read requirement: normalize(None) must return ""; strings should be stripped and lowercased.
First patch: if text is None: return None. This preserves the wrong missing-value behavior.
Run tests: The missing-value assertion fails. This test call reveals the defect; the call itself is not an error.
Repair: Change the missing-value return to "". Keep string handling as text.strip().lower().
Verify final state: Run the required tests on the final patch. Grade their results, not the agent’s claim of success.

A trace can show that the agent read the right file, reproduced a failure, edited a relevant line, or interpreted a tool result incorrectly. These observations are useful for debugging even when the final task fails.

Avoid giving points simply for more actions. Repeatedly opening the same file should not increase progress. Define a small set of independently checkable milestones and count each at most once. Allow valid alternative routes rather than requiring a single reference trajectory.

Outcome and diagnostic checks answer different questions

Constructed runs.

Run A: Final tests fail. The agent located the relevant file and reproduced the failure: useful progress, incomplete task.
Run B: Final tests pass. The agent found a valid patch through a different sequence of actions: complete task.
A stable diagnostic rubric: Record independently checkable milestones once each. Repeating a read or test call must not create more credit.
Keep separate outputs: Report final success and diagnostic observations separately. A high partial score does not turn a failed task into success.

A diagnostic milestone can also depend on the evaluator. “Ran tests” is easy to observe; “verified the fix” requires knowing that the relevant tests ran successfully against the final state. Keep that distinction in the labels.

Reliability takes repeated trials

For $n$ attempts with $k$ successes, the observed pass rate is $\hat p=k/n$ . Under independent Bernoulli sampling with a common success probability, an approximate 95% Wilson interval uses $z=1.96$ :

\text{center}=\frac{\hat p+z^2/(2n)}{1+z^2/n},\qquad \text{half-width}=\frac{z\sqrt{\hat p(1-\hat p)/n+z^2/(4n^2)}}{1+z^2/n}.

The interval is center plus or minus half-width. Its center is not generally $\hat p$ . For six successes in ten attempts it is about 31.3% to 83.2%, which conveys how little ten runs establish.

Repeated attempts estimate reliability

Seeded Bernoulli toy simulation with assumed success probability 0.60. These are not agent measurements.

Attempts included: 10

8/10 successful: 80.0%. Approximate 95% Wilson interval: 49.0%–94.3%.

1:pass2:pass3:pass4:pass5:pass6:pass7:pass8:fail9:fail10:pass

Real repeated runs may share task difficulty and environmental failures. Independence must be justified, not inferred from different random seeds.

Real tasks differ in difficulty, and trials on the same task share that difficulty. Pooling them as independent identical trials can understate uncertainty or answer the wrong question. Preserve task identifiers, compare systems on the same tasks, and use an analysis that respects the sampling design, such as resampling tasks rather than treating every repeated attempt as unrelated.

Two repeated-trial metrics ask opposite questions. pass@k counts whether at least one of $k$ attempts succeeds. pass $^k$ , used by $\tau$ -bench for reliability, asks whether all $k$ succeed. Under the simple independent fixed- $p$ model they are $1-(1-p)^k$ and $p^k$ , respectively. Average per-task values when task success probabilities differ; applying either formula to an aggregate mean generally gives a different answer.

Any success and consistent success are different metrics

Constructed independent attempts on one task with a fixed per-attempt success probability.

Per-attempt success: 0.60Attempts k: 3

At least one success, pass@k: 1 − (1 − p)^k = 93.6%.

Every attempt succeeds, pass^k: p^k = 21.6%.

For heterogeneous tasks, apply these formulas per task before averaging. A human comparison also needs the same task set, tools, time limit, and success criterion.

Control the environment, then record what remains variable

A fixed random seed does not reset a repository, database, browser session, clock, or external service. Save the starting state and reset it between runs. Record tool versions, timeout rules, retry behavior, permissions, and any network dependencies.

WebArena provides self-hosted websites for controlled browser tasks. In evaluations that instead use live services, record changes to those services as part of the measurement.

Deterministic-looking operations can still reveal hidden dependencies. A program that relies on the order returned by an unordered directory listing, for example, can behave differently across environments.

Compare quality under a budget

Report cost per attempt and latency together with success. Include failed attempts, tool usage, and any candidate selection. A cheaper unsuccessful run can look attractive if the report counts only its model tokens and omits the work needed to recover.

Spend more only when it buys the measured outcome

Constructed systems with assumed mean cost per attempt and success probability. No uncertainty is represented.

Cost limit per attempt: 5 units

A: 1 units; 40% success; Pareto efficient.
B: 2 units; 60% success; Pareto efficient.
C: 3 units; 50% success; dominated.
D: 5 units; 80% success; Pareto efficient.
E: 8 units; 80% success; dominated; over budget.

Highest assumed success within this budget: D. A system is dominated if another costs no more and succeeds at least as often, with a strict improvement in one dimension.

A system is Pareto dominated when another costs no more and succeeds at least as often, with a strict improvement in one of those dimensions. The example above assumes its numbers are known; real comparisons also need uncertainty and comparable cost accounting.

For identical independent restarts on one task, with success probability $p>0$ , constant cost $c$ per attempt, and a reliable way to recognize success, expected cost until the first success is $c/p$ . Correlated retries, changing strategies, imperfect verification, and tasks with different $p$ values change the interpretation. In particular, averaging $c_i/p_i$ across tasks is not the same as dividing mean cost by mean success probability.

For a human baseline, report participants’ expertise, tools, time budget, task selection, and success criterion. A human score under one protocol is neither a universal ceiling nor directly comparable to an agent allowed unlimited retries.

A minimal loop with an explicit evaluator

This sketch leaves the model and tools behind interfaces so the evaluation contract is visible. step returns a structured tool event or a finish event. The environment evaluator runs after either completion or budget exhaustion, and its result must be a boolean. Milestones come from trusted evaluation code, not from the agent’s self-report.

A malformed agent event is an execution error and still reaches final grading. A production harness also needs environment resets, timeouts, resource accounting, and durable traces. Evaluator failures should remain visible as evaluation errors, rather than being silently converted into successes or omitted from the report.

The human preference evaluation post turns to tasks where people’s judgments are part of the intended outcome.

Additional reading

SWE-bench. Task definitions, benchmark variants, and evaluation resources.
Yao et al., $\tau$ -bench. Tool-agent-user interaction, final-state evaluation, and pass $^k$ reliability.
Zhou et al., WebArena. Controlled self-hosted web environments and functional task evaluation.
NIST, confidence limits for a proportion. Interval estimation for binary outcomes.