Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation
Across 496 expert-reviewed tasks on BFCL v4, tau2-Bench, LiveMCPBench and MCP-Atlas: 92 evaluator-human disagreements (18.5% misalignment). In LiveMCPBench, 23 repeats of the SAME setup score 57.9%-76.8% — an 18.9-point spread, wide enough to reorder a leaderboard. Taxonomy of deterministic vs LLM-judge failure families.
Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation
Abstract
Verbatim from the arXiv abstract page (fetched 2026-09-11):
Tool-calling benchmarks are increasingly used to rank language-model agents, yet their scores are often treated as ground truth without validating the evaluators themselves. We present a systematic validity and reproducibility audit of four major tool-calling benchmark families: BFCL v4, τ2-Bench, LiveMCPBench, and MCP-Atlas. Across 496 expert-reviewed benchmark tasks, we find 92 evaluator-human disagreements, corresponding to an 18.5% misalignment rate. The failures are not isolated annotation mistakes: deterministic benchmarks exhibit brittle state matching, trajectory lock-in, incorrect ground truths, substring-based communication failures, and reward-basis misalignment, while LLM-judge benchmarks exhibit rubric drift, hallucinated completion, answer-only scoring, and substantial run-to-run variance. In LiveMCPBench, 23 repeated evaluations of the same setup produce scores ranging from 57.9% to 76.8%, a spread of 18.9 percentage points, large enough to change leaderboard conclusions. These results show that current tool-calling scores can reflect evaluator artifacts rather than agent capability. We introduce a unified taxonomy of tool-calling evaluation failures, release trace-level audit artifacts and corrected evaluation components, and argue for decomposed metrics that separately measure tool invocation, task completion, and outcome verification. Our findings suggest that progress in tool-using agents requires benchmarks whose evaluators are themselves reproducible, auditable, and aligned with human judgments of task success. We further introduce Tool-Veritas, a configurable benchmark that combines deterministic state verification with optional qualitative judging, and Harness Lab, an open-source system for benchmark execution, trace inspection, repeated-run comparison, and evaluator debugging.
Key Contributions
- The evaluator is the thing being audited, not the agent. Four benchmark families in scope: BFCL v4, τ²-Bench, LiveMCPBench, MCP-Atlas.
- A measured misalignment rate. 92 evaluator-human disagreements over 496 expert-reviewed tasks = 18.5%.
- A measured noise floor. 23 repeated runs of an identical LiveMCPBench setup produce 57.9% to 76.8% — an 18.9-point spread. This is the number that matters: the run-to-run variance of the instrument exceeds the gaps between models that leaderboards report as findings.
- A taxonomy of failure families, split by evaluator kind:
- Deterministic evaluators — brittle state matching, trajectory lock-in (only one accepted path), incorrect ground truths, substring-based communication checks, reward-basis misalignment.
- LLM-judge evaluators — rubric drift, hallucinated completion, answer-only scoring (grading the final string, not the trajectory), run-to-run variance.
- A prescription: decompose the metric into tool invocation, task completion, and outcome verification, scored separately, rather than one headline pass rate.
- Two artifacts released: Tool-Veritas (deterministic state verification + optional qualitative judge) and Harness Lab (execution, trace inspection, repeated-run comparison, evaluator debugging).
Methodology
Expert human review of 496 benchmark tasks drawn across the four families, with each task's evaluator verdict compared against the human verdict; disagreements are then classified into the taxonomy rather than merely counted. Separately, a single LiveMCPBench configuration is re-run 23 times to characterise the variance of the evaluator itself with the agent and setup held fixed.
Results
| Measurement | Value |
|---|---|
| Expert-reviewed tasks | 496 |
| Evaluator-human disagreements | 92 |
| Misalignment rate | 18.5% |
| LiveMCPBench repeated runs, same setup | 23 |
| Score range across those runs | 57.9% – 76.8% |
| Spread | 18.9 percentage points |
Limitations
- The 496 audited tasks are a sample across four families; the 18.5% figure is a rate on that sample, not a population rate for tool-calling evaluation generally.
- The 18.9-point spread is measured on LiveMCPBench specifically — an LLM-judge benchmark with a live environment. It is not evidence that deterministic benchmarks carry the same variance, and the paper's own taxonomy separates the two failure classes.
- Tool-Veritas and Harness Lab are the authors' own instruments; the paper is both the audit and the proposed replacement, which is the standard conflict in this genre.
- Expert review is itself a judgment; "evaluator-human disagreement" assumes the human is right, which the paper asserts rather than independently validates.
Why this is on the evals rung
The rung's founding question is whether a measurement can be trusted. This is the first source it holds that puts a number on the instrument rather than on a model: a benchmark whose run-to-run noise (18.9 points) exceeds the margins it reports is not measuring. That is checkable, teachable and falsifiable — anyone can re-run a leaderboard 23 times.
No live market call rests on this rung; this is a brand/teaching source, taken as the capped no-bet lane for the cycle.
Source: Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation by Vishvesh Bhat, Jay Vaghasiya, Muhammad Ahmed Mohsin, Asad Aali — arXiv:2607.02577, submitted 2026-06-30.