Tool and Agent Evaluation

The previous posts in this arc covered static evaluation: give a model a question, get an answer, compare to a ground truth. The machinery is comparatively clean. You pick a benchmark, you score outputs, you maybe worry about contamination or prompt sensitivity. But the output is a string, and a string you can compare.

Agent evaluation breaks all of that. The output of an agent run isn't a single string. It's a whole trajectory of decisions and environment responses strung together, and any scoring scheme has to grapple with the whole path.

When an agent is told to fix a bug, it makes a series of decisions: which file to read first, which edit to attempt, whether to run tests before or after the fix, how to interpret test failures and retry. Each of those decisions compounds into the next. The final outcome (did the bug get fixed, did the test suite pass) depends on everything that came before it. Scoring just the output misses most of what matters.

I want to build up a working mental model for this layer of the evaluation stack.

Why agent eval is structurally different

The difference isn't just that agents are harder to evaluate. The structure of the problem is different in a few specific ways.

Outputs are trajectories, not answers. A static benchmark produces one token sequence and scores it. An agent produces a sequence of (action, observation) pairs that can stretch to dozens of steps and branch based on what the environment returns. The "output" is the whole path, not just the destination.

The environment is stateful. When an agent reads a file, that reading happened. When it edits a file, the edit persists. If the agent runs tests and a test fails, the agent now has to reason about that failure and decide what to do next. None of this is captured in a single forward pass. The trajectory is partly determined by the model and partly determined by everything the environment fed back to it along the way.

Outputs are non-deterministic. Run the same agent on the same task twice and you'll get different trajectories. Temperature-based sampling means the model makes different decisions at each step, and small divergences early in the trajectory compound into large ones by the end. Unlike a static benchmark where the model either knows the answer or doesn't, an agent might fix a bug 40% of the time and fail it 60% of the time just due to sampling noise. You need enough runs to get statistically stable estimates.

There's no universal ground truth for the path. There are often many valid trajectories to a correct solution. The agent might read three different files in a different order, apply a slightly different patch, and still pass the tests. The path doesn't need to be canonical; the outcome does. But partial credit for the path is still meaningful, as we'll get to.

What you need in your head to follow along

A few handles carry most of the weight here. Two of them actually belong to arc 10, which is a fair way ahead of where you are if you're reading in order. I'd rather define them in a sentence each right now than send you jumping forward and back, so here's the short version. The full treatment is waiting for you in arc 10.

The agent loop (arc 10): model proposes an action, the environment returns an observation, both get appended to context, repeat until a stop condition fires. Every "trajectory" in this post is a sequence of those (action, observation) pairs. That sentence is genuinely all you need to follow this post.

Function calling (arc 10): the action is almost always a structured call into a named tool (read_file, bash, http_get) with JSON arguments. The environment's response is a string or a structured result. Tool reliability is a separate concern handled back in arc 7; here we assume the tool works and focus on whether the agent drove it well.

Evaluating reasoning: pass rates, contamination, prompt sensitivity. Agent eval inherits all of these and adds the trajectory problem on top.

One new bit of stats we'll lean on: a pass rate is a proportion from a binomial sample. Treat it as an estimate with a confidence interval, not a number.

The math you actually need

There are three formulas that do most of the work. None of them are deep; they just need to be written down somewhere so you can reach for them.

Trajectory score. Given oracle checkpoints $c_1, \ldots, c_K$ a good agent should hit, and an indicator $\mathbb{1}[c_k]$ that's 1 when step $k$ was hit,

Pass rate is $\mathbb{1}[\text{final outcome ok}]$. Trajectory score is everything in between.

Confidence interval on a pass rate. With $k$ passes out of $n$ runs, the Wilson interval is the one you want for small $n$ (it behaves sensibly when $k = 0$ or $k = n$):

with $z = 1.96$ for 95%. Plug in $k = 6$, $n = 10$ and the interval is roughly $[31\%, 83\%]$. That's the actual range ten runs pins down. It's wide.

Cost per success. If the agent passes with probability $p$ at a token cost $c$ per attempt (and you let it retry until it passes),

This is the number that decides whether a cheap-and-mediocre agent beats an expensive-and-good one in production. We'll use it again in the cost section.

End-to-end pass rate

The most widely used metric in agent evaluation is end-to-end pass rate: what fraction of tasks does the agent fully complete?

SWE-bench, the most influential agent benchmark in the software-engineering domain, defines completion as: the agent's patch is applied to the repo, and the hidden test suite passes. The agent can take any number of steps, use any tools it has available, and read any files it wants. If the tests pass when it's done, that's a pass. If they don't, that's a fail.

This is a good metric because it's concrete, unambiguous, and hard to game. The tests are held out. The agent doesn't see them. Either it fixed the bug well enough to pass them, or it didn't.

But it's a blunt instrument. It tells you whether the agent solved the problem. It doesn't tell you how close it got. An agent that correctly identified the file, correctly located the bug, and then applied a syntactically wrong patch would score zero. So would an agent that hallucinated a completely wrong file path from the start. Those are very different failure modes, and zero-vs-zero tells you nothing about the difference.

SWE-bench Verified addresses a different problem. Not the bluntness problem, the contamination problem. The original SWE-bench tasks come from public GitHub issues that many models have seen in training. Verified is a human-curated subset where OpenAI (in collaboration with the SWE-bench team) had software developers manually confirm that the tasks are solvable, that the reference solution is correct, and that the test setup is clean. It's 500 tasks instead of 2294, but the 500 are trustworthy. Most serious comparisons now use Verified.

SWE-bench scores as of early 2026: frontier models like Claude Opus and GPT-4o agents are in the 40–55% range on SWE-bench Verified. Human expert programmers, given the same task descriptions but no time constraints, score around 75%. That gap is the real distance between current agents and human-level coding ability on these tasks.

Trajectory scoring: partial credit that actually matters

End-to-end pass rate is the right top-line metric. But if you're trying to make your agent better, you need to know more than pass/fail. You need to know where in the trajectory things go wrong.

Trajectory scoring gives partial credit for correct intermediate actions. The idea is to define a sequence of checkpoints a good agent should hit (opened the right file, called the right function, produced a patch that at least compiles), and score each one independently. An agent that gets 5 out of 7 checkpoints right before failing the final test has a higher trajectory score than an agent that goes completely off the rails at step 1.

A 7-step SWE-bench-style trajectory. Step 4 fails (wrong strip order), the agent recovers at step 6, tests pass at step 7. End-to-end: PASS. Trajectory: 6/7.

The payoff is visible in the step-through above. When the agent makes a wrong move in the middle (bad test run, wrong operand order) but recovers and fixes it, the end-to-end outcome is PASS but the trajectory score is 6/7. Knowing that the agent failed on exactly one step type is the signal you need to diagnose and improve. Without trajectory scoring, a run that ends in PASS looks the same as a run that got lucky on every step.

This is why trajectory scoring matters for making progress visible even on hard benchmarks where end-to-end pass rates are low. If your baseline agent passes 20% of tasks end-to-end, but you make changes that improve trajectory scores from 45% to 60%, you've likely made the agent meaningfully better even if the pass rate didn't move yet. Pass rate is a lagging signal; trajectory score is a leading one.

The practical challenge is that trajectory scoring requires either human annotation (expensive) or an oracle that knows the correct intermediate steps (brittle). Most published benchmarks only report end-to-end pass rates because trajectory scoring is operationally hard. tau-bench is one of the few benchmarks that attempts systematic trajectory evaluation in a realistic production setting (tool-use in customer service workflows), and it's worth studying for how they handle the annotation problem.

Four agents with similar end-to-end pass rates but very different trajectory scores. Pass rate is a lagging signal; trajectory score surfaces which agent is following the right strategy.

The chart above shows why end-to-end alone can be misleading. Agents that look similar on pass rate are far more differentiated on trajectory score. If you're trying to pick between systems or diagnose regressions, trajectory score is the metric you want.

Environment determinism is harder than it looks

Here's a scenario that comes up once you've run agent evals at any scale.

You have a task. You run your agent on it ten times with the same model, same tools, same prompt. It passes six times and fails four times. Is the agent good enough for this task? Depends on what you care about. But more concretely: which of those six passing runs should you analyze? And how do you know the four failures are real failures rather than environment noise?

A single agent run is a sample drawn from a noisy distribution. Treating it as a verdict about the agent can easily convince you a model is better or worse than it actually is. Until you've run enough trials to narrow the confidence interval, you don't actually know what the pass rate is. Ten runs can easily miss the true rate by 15 points. Thirty is more defensible. Most published agent scores are at 100 runs per task or more, and even that is often not enough for fine-grained comparisons between models whose true rates are close.

Environment non-determinism comes from two places.

The first is the model itself. Sampling is stochastic. Different runs of the same prompt will produce different action sequences. This is expected and manageable with enough samples.

The second is the environment, which is where it gets messier. The output of ls isn't guaranteed to be in the same order on every run (filesystems have varying directory entry orderings). A test suite might have timing-dependent flakes that cause intermittent failures unrelated to the agent's patch. An API the agent calls might be rate-limited, or might return slightly different error messages depending on load. A tool that scrapes a website will get different content on different days.

For internal SWE-bench-style evaluations, most of these problems are manageable because the test environments are controlled Docker containers with fixed state. But for production agent evaluations against real systems (WebArena, OSWorld), environment non-determinism is a first-class problem. The same agent browsing the same website on different days will encounter different page states.

The "can't replay" problem is related. As models are updated and tools evolve, historical trajectories stop being reproducible. A trajectory log from six months ago was generated by a different model version, against a tool API that may have changed, in an environment that may have drifted. You can't re-run it to verify whether the scoring was right. This makes long-term trend analysis genuinely difficult. The field is still figuring out how to handle this.

Cost-normalized evaluation

Let me add one more dimension that I think is underweighted in most published benchmarks: cost.

An agent that passes 95% of tasks while spending 50x more tokens than an agent that passes 90% of tasks is not obviously better. Depending on your application, the 90%-at-1x agent might be strictly preferable. Or the 95%-at-50x agent might be fine if you only need to run it once a day. The right answer is "it depends," but the relevant information for making that call (pass rate AND cost per task) is almost never reported together.

Each dot is an agent. The frontier is the set you can't beat on both axes at once. Agents below the line are dominated; the right pick depends on whether your constraint is cost or pass rate.

The Pareto frontier framing is useful here. For a fixed benchmark, you can plot each agent as a point in (cost, pass rate) space. The Pareto frontier is the set of agents that can't be simultaneously beaten on both axes. Agents below the frontier are dominated: some other agent is cheaper AND gets better results. The only meaningful choice is a point on the frontier, and the right point depends on your production constraints.

Anthropic's internal system card evaluations for Claude models now report performance at multiple operating points (different context budgets, different tool limits). OpenAI does similar for GPT-4o with different tool-use configurations. This is good practice, but it's still the exception rather than the norm in published benchmarks.

A quick worked example. If Agent A costs 10k tokens per task and passes 40% of tasks, and Agent B costs 100k tokens per task and passes 45% of tasks:

• Agent A's cost-per-success: 25k tokens.
• Agent B's cost-per-success: ~222k tokens.

Agent B's pass rate is 12.5% higher, but Agent A's cost-per-success is 9x lower. Whether that trade-off is worth it depends entirely on what you're trying to do.

Human baselines and calibration

Human baselines matter for calibration. Without them, "40% pass rate" is just a number. It doesn't tell you whether that's impressively good or a long way from useful.

For SWE-bench, the human baseline was established by having experienced software engineers attempt the same tasks with the same constraint (no knowledge of the hidden test suite, access to the same codebase state). The result was approximately 75% pass rate. That's the ceiling current frontier agents are working toward.

The gap at any moment in time is interpretable: a 55% agent solves about three-quarters of what a human expert would solve. You can have concrete conversations about where the remaining failures are concentrated (multi-file changes? test-writing tasks? tasks requiring deep domain knowledge?) in a way that "55% is pretty good" doesn't support.

Voyager, the open-ended Minecraft agent benchmark, took a different approach to baselines. Since there's no human expert to run on the Minecraft task distribution, the authors compared against progressively stronger agents (no feedback, with feedback, with skill library). The relative baselines still give calibration even without an absolute human ceiling.

GAIA is a useful contrast. GAIA is an agent benchmark specifically designed to be easy for humans (92% human solve rate) but hard for models (frontier models score around 55–65% as of early 2026). It deliberately inverts the typical situation: the human baseline tells you the floor for what a useful agent should achieve, not the ceiling. Tasks that are trivially easy for humans but fail for models reveal something qualitatively different from tasks that are hard for everyone.

A minimal eval harness

Here's what a lightweight trajectory-scoring harness looks like in practice. The structure is simple: run the agent, log the trace, compare against oracle checkpoints.

from dataclasses import dataclass, field
from typing import Callable, Any
 
@dataclass
class Step:
    tool: str
    args: dict
    result: Any
    checkpoint_hit: bool = False
 
@dataclass
class TrajectoryResult:
    task_id: str
    steps: list[Step] = field(default_factory=list)
    final_pass: bool = False
 
    @property
    def trajectory_score(self) -> float:
        """Fraction of oracle checkpoints hit, regardless of final outcome."""
        if not self.steps:
            return 0.0
        hits = sum(1 for s in self.steps if s.checkpoint_hit)
        return hits / len(self.steps)
 
    def summary(self) -> str:
        n = len(self.steps)
        hits = sum(1 for s in self.steps if s.checkpoint_hit)
        return (
            f"task={self.task_id}  pass={self.final_pass}  "
            f"traj={hits}/{n} ({self.trajectory_score:.0%})"
        )
 
def score_trajectory(
    agent_fn: Callable[[str, list[Step]], dict],
    env_fn: Callable[[str, dict], Any],
    oracle_fn: Callable[[Step], bool],
    task: str,
    max_steps: int = 20,
) -> TrajectoryResult:
    """
    agent_fn(task, history) -> {"tool": ..., "args": ...}
    env_fn(tool, args)       -> observation (side-effecting)
    oracle_fn(step)          -> True if this step hit a checkpoint
    """
    result = TrajectoryResult(task_id=task)
    history: list[Step] = []
 
    for _ in range(max_steps):
        action = agent_fn(task, history)
        if action.get("tool") == "finish":
            result.final_pass = bool(env_fn("run_tests", {}))
            break
 
        observation = env_fn(action["tool"], action.get("args", {}))
        step = Step(
            tool=action["tool"],
            args=action.get("args", {}),
            result=observation,
            checkpoint_hit=oracle_fn(
                Step(tool=action["tool"], args=action.get("args", {}), result=observation)
            ),
        )
        history.append(step)
        result.steps = history
 
    return result
 
# Oracle for a null-ptr bug task
def null_ptr_oracle(step: Step) -> bool:
    # Hit if agent reads the right file
    if step.tool == "read_file" and "normalizer" in str(step.args):
        return True
    # Hit if agent's patch targets the None check
    if step.tool == "edit_file" and "None" in str(step.args):
        return True
    # Hit if agent runs tests at least once
    if step.tool == "bash" and "pytest" in str(step.args):
        return True
    return False

The oracle function is the part that requires thought. For SWE-bench-style tasks, you'd define checkpoints based on the reference solution: did the agent touch the right file? Did it produce a patch that at least addresses the right function? You can define these programmatically from the ground truth solution rather than having to annotate each trajectory by hand.

Benchmarks worth knowing

A quick map of the landscape, since names come up constantly and it helps to have mental handles.

SWE-bench / SWE-bench Verified: The canonical software-engineering agent benchmark. Real GitHub issues, real test suites. The Verified subset is the one to use for serious comparison. Leaderboard maintained at swebench.com.

SWE-agent: Both a specific agent scaffold (GitHub agent with file viewer, search tools, patch application) and a paper analyzing what makes agents succeed on SWE-bench tasks. Useful for understanding why the default tool set for coding agents looks the way it does.

tau-bench: Tool use in realistic production API workflows (airline reservations, retail transactions). Focuses on multi-step reasoning with structured APIs rather than free-form coding. Harder to game than coding benchmarks and closer to production conditions.

WebArena: Web navigation tasks (shopping, forum posts, code repository interaction) in a sandboxed browser. Non-deterministic by nature. Harder to reproduce than SWE-bench but more representative of real-world agent deployments.

OSWorld: Computer-use tasks in full desktop environments. Screenshot-based observation, click-and-type actions. The most realistic setting for GUI agents, and correspondingly the hardest.

GAIA: General AI assistant tasks calibrated to be easy for humans, hard for models. Good for identifying specific failure modes rather than measuring overall capability.

Voyager: Open-ended task completion in Minecraft. Useful for studying agents that build skills over time rather than solve isolated tasks.

Misconceptions

"High SWE-bench scores mean the agent is production-ready." SWE-bench Verified is 500 carefully selected tasks with clean test environments, fixed codebases, and no time pressure. Production code agents operate on messy repos with flaky tests, unclear requirements, broken CI, and real-world API rate limits. The benchmark is a signal, not a guarantee. An agent at 55% on Verified might underperform badly on your specific codebase if your test setup is messier or your issue descriptions are ambiguous.

"Trajectory scoring is just a softer version of pass rate." They measure different things. Pass rate asks "did the agent solve the problem?" Trajectory score asks "was the agent following the right strategy?" An agent can score well on trajectory and fail the task (good process, unlucky outcome) or score poorly on trajectory and pass (got lucky with a brute-force approach). They're complementary, not redundant.

"Non-determinism means you can't trust a single run." You can't trust it as a statistical estimate, but you can often trust it as a debugging signal. If your agent fails a task three times in a row in exactly the same way, you're looking at a systematic issue rather than noise. Conversely, if an agent passes a task once but the trajectory looks fragile, that's worth investigating even if the pass happened. Treat one run as a sample to inspect, not as a score to report.

What's next

The next post covers human preference evaluation, the other leg of the evaluation stack: when there's no ground truth to compare against and you need to know whether people actually prefer one model's output over another. RLHF, Chatbot Arena, and the challenges of converting human opinions into stable training signals.

Additional reading (and watching)

• Jimenez, C. E., et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024. Introduced the benchmark, established the human expert baseline at ~75%, and analyzed why models fail on multi-file and integration-test tasks.

• SWE-bench Leaderboard. (2026). https://www.swebench.com. Live leaderboard tracking agent performance on SWE-bench Verified. Current frontier models are in the 40–55% range as of early 2026.

• Yao, S., et al. (2024). tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Structured API tool-use with state consistency requirements; closer to production conditions than coding benchmarks.

• Zhou, S., et al. (2024). WebArena: A Realistic Web Environment for Building Autonomous Agents. ICLR 2024. Sandboxed browser agent tasks across shopping, forums, and code repos — non-deterministic environments.

• Xie, T., et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurIPS 2024. Desktop computer-use tasks with full GUI access — the most realistic (and hardest) agent benchmark currently available.

• Wang, G., et al. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. The Minecraft agent that builds skills over time; shows how to construct relative baselines when no human ceiling applies.

• Mialon, G., et al. (2023). GAIA: A Benchmark for General AI Assistants. ICLR 2024. Calibrated for high human solve rates (~92%) to measure how far models fall short on tasks humans find trivial.

• Yang, J., et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS 2024. Both an agent scaffold and an analysis of what tool interfaces enable successful code agents.