AI evals are becoming the new compute bottleneck

How deeply this was read: the full article text, fetched from the Hugging Face page on 2026-09-24 (published 2026-04-29, "Community Article"). Every figure below is copied from that text. The article is a blog post, not a peer-reviewed paper. Most of its numbers are the authors adding up costs that other papers and the live Holistic Agent Leaderboard (HAL) report; they are compilations, not new experiments. It is filed as analysis for that reason.

The question it answers for this rung: what does it cost to measure a model or an agent at all — and does that cost stop people checking the numbers everyone else quotes?

Abstract

The article's own summary, verbatim:

AI evaluation has crossed a cost threshold that changes who can do it. The Holistic Agent Leaderboard (HAL) recently spent about $40,000 to run 21,730 agent rollouts across 9 models and 9 benchmarks. A single GAIA run on a frontier model can cost $2,829 before caching. Exgentic's $22,000 sweep across agent configurations found a 33× cost spread on identical tasks, isolating scaffold choice as a first-order cost driver, and UK-AISI recently scaled agentic steps into the millions to study inference-time compute. In scientific ML, The Well costs about 960 H100-hours to evaluate one new architecture and 3,840 H100-hours for a full four-baseline sweep. While compression techniques have been proposed for static benchmarks, new agent benchmarks are noisy, scaffold-sensitive, and only partly compressible. Training-in-the-loop benchmarks are expensive by construction, and when you try to add reliability to these evals, repeated runs further multiply the cost.

Key Contributions

• A price list for measuring. One table normalises 16 benchmarks to "US dollars per single evaluation", converting GPU time at $2.50 per H100-hour and $1.50 per A10-hour. The cheapest single run in it is a few cents; the most expensive single benchmark runs about $9,500 (PaperBench, full protocol); the HAL total is about $40,000.
• The scaffold is a cost driver, not a detail. A $22,000 sweep across agent set-ups (Exgentic) found a 33× cost spread on identical tasks. The article's own summary: agent benchmarks "benchmark a model × scaffold × token-budget product, and small scaffold choices can multiply costs 10×."
• Spending more did not buy a better score. On Online Mind2Web, one agent cost $1,577 for 40% accuracy and another $171 for 42% — HAL's paper calls this "a 9× difference in cost despite just a two-percentage-point difference in accuracy." On GAIA, one set-up cost $2,828 for 28.5% and another $1,686 for 57.6%.
• The shortcuts stop working as benchmarks get closer to real work. Static benchmarks can be cut 100–200× (HELM) and still rank models the same way. Agent benchmarks compress only 2–3.5× (a filter that keeps tasks with 30–70% historical pass rates). Benchmarks that train a model as part of the test have no general shortcut at all.
• Reliability is the multiplier. Re-running every HAL cell 8 times to get a trustworthy number takes the ~$40,000 to roughly $320,000.

Methodology

Not an experiment. The authors collect cost figures from published benchmark papers (HELM, PaperBench, MLE-Bench, RE-Bench, ResearchGym, The Well, others) and from the live HAL leaderboard (April 2026), then put them on one scale: dollars for one model or agent through one full benchmark, with GPU hours converted at the two fixed hourly rates above. Where a paper reports only GPU hours, the dollar figure is the authors' conversion, not a receipt.

Results

Per-benchmark spread on HAL (min to max cost for one full run): SWE-bench Verified Mini $4–$1,600; Online Mind2Web $5–$1,610; GAIA $7.80–$2,829; TAU-bench Airline $0.31–$180.

Limitations

• A blog post by an advocacy group. The EvalEval Coalition runs "Every Eval Ever", a shared format for eval results, and the article ends by recommending it. The cost numbers do not depend on that recommendation, but the framing does.
• Mostly other people's numbers. Figures are gathered from papers and a live leaderboard, then converted at fixed GPU rates. The authors say the figures are "lower bounds".
• The 33× is not the same measurement as HAR-1's 40×. The 33× is a dollar spread across agent configurations on the same tasks (Exgentic); the harnesses rung's 40× is a tokens per solved task spread across three harnesses with the model fixed (The Scaffold Effect in Coding Agents). They point the same way; they are not two readings of one number.
• Prices are dated. API prices quoted (e.g. Claude Opus 4.1 at $15/$75 per million input/output tokens) are the article's, as of April 2026.
• The ~$320,000 is arithmetic (8 × the single-seed total), not a run anyone paid for.

How it bears on the through-line

The KB tracks the path from a token price to a task price. Every task price rests on a benchmark score, and this article prices the score itself. Two consequences. First, the scaffold moves the cost of a finished task by an order of magnitude with the model held still — the same lever the harnesses rung measures, seen from the evaluator's side. Second, a trustworthy score costs about 8× a single-run score, so most published scores are single runs, and a leaderboard that lists accuracy without cost rewards whoever spent the most tokens. Whoever can pay for the evaluation writes the leaderboard.

Source: AI evals are becoming the new compute bottleneck by Avijit Ghosh, Yifan Mai, Georgia Channing and Leshem Choshen, EvalEval Coalition, 2026-04-29.