The Cost of Evaluation

Measuring a model used to be the cheap part. For agents it no longer is. Running one public agent leaderboard — nine models on nine benchmarks, each once — cost about $40,000 for 21,730 runs, and a single run of one agent through one benchmark (GAIA) can cost $2,829 before caching (AI evals are becoming the new compute bottleneck). Those are single runs. A single run of an agent is a noisy reading: in one tool-calling benchmark, 23 repeats of the same set-up scored anywhere from 57.9% to 76.8% (Benchmarking the Benchmarks). Getting a number you can trust means running each cell several times, and at 8 repeats the $40,000 leaderboard becomes roughly $320,000.

So there are two prices for a benchmark score: the price of a quick reading, and the price of a trustworthy one, about 8× apart. Almost every published score is the quick reading.

Why agents broke the old shortcuts

For fixed question-and-answer benchmarks, most of the questions turn out not to matter for the ranking: one study cut HELM's compute 100–200× and got nearly the same ordering of models. For agent benchmarks the same trick saves only 2–3.5×, because each "question" is a long, variable run rather than one answer. For benchmarks that train a model as part of the test, there is no general shortcut at all (AI evals are becoming the new compute bottleneck).

The scaffold is a cost driver

A $22,000 sweep across agent set-ups found a 33× cost spread on identical tasks — the model is only one part of what is being paid for; the scaffold around it and the token budget are the rest. And spending more did not reliably buy a better score: on one web benchmark, $1,577 bought 40% and $171 bought 42% (AI evals are becoming the new compute bottleneck). The harnesses rung measures the same effect with the model held fixed — up to 40× the tokens per solved task from the harness alone (The Scaffold Effect in Coding Agents). The two numbers are different measurements (dollars across set-ups; tokens across harnesses) that point the same way.

This is also why budget matters to the score, not just the bill: fixed single-budget evaluations understate newer models, whose best results only appear at generous budgets (How Inference Compute Shapes Frontier LLM Evaluation, abstract level). A score and its spend are one measurement.

Key Claims

• One agent leaderboard, one run per cell: ~$40,000 (21,730 rollouts, 9 × 9). An independent reproduction reached ~$46,000. Evidence: moderate (blog compilation of the live leaderboard) (EvalEval)
• Scaffold choice alone spreads cost 33× on identical tasks. Evidence: moderate (reported second-hand from the Exgentic sweep) (EvalEval)
• Reliability multiplies every cost: 8 repeats take the leaderboard to ~$320,000. Arithmetic, not a paid run. Evidence: moderate (EvalEval)
• One run is not a reading: 23 repeats of the same set-up spanned 57.9%–76.8%. On LiveMCPBench, an LLM-judged benchmark. Evidence: strong for that benchmark (Benchmarking the Benchmarks)
• Compression works for static benchmarks (100–200×) and barely for agents (2–3.5×). Evidence: moderate (EvalEval)

Benchmarks & Data

Benchmark	Cost of one evaluation (as reported)
TAU-bench Airline	$0.31 – $180
SWE-bench Verified Mini	$4 – $1,600
Online Mind2Web	$5 – $1,610
GAIA	$7.80 – $2,829
PaperBench, full	~$9,500
HAL, 81 cells, single seed	~$40,000

GPU time converted at $2.50/H100-hour and $1.50/A10-hour by the article's authors; as of April 2026 (EvalEval).

Open Questions

• What does a trustworthy score cost on the benchmarks vendors actually quote — how many repeats does each need before the spread is smaller than the gaps it reports?
• Does sharing run logs (the article's proposed fix) actually cut what the field spends, or only what it could spend?
• Nobody has put a cost column on a major leaderboard's headline table; until someone does, a ranking cannot be read as a price.

Related Concepts

• Benchmark Validity — whether the score measures the thing named; this page is what it costs to find out.
• Agent Evaluation Benchmarks — the harness, budget and held-out standard a result must disclose.

Changelog

• 2026-09-24 — Created from the EvalEval cost article (ingested today), with the LiveMCPBench repeat-run spread and the inference-compute paper folded in.