Benchmark Validity

A benchmark score makes two promises: that the grader marked the answer correctly, and that the benchmark measures the ability it is named after. Both can fail, and on this rung both have been measured failing.

The grader can be wrong. An audit of four tool-calling benchmark families checked 496 tasks by hand and found the benchmark's own grader disagreed with the experts on 92 of them — an 18.5% error rate. The failures came in two families: rule-based graders that only accept one exact path or string, and AI-judge graders that drift, mark unfinished work as done, or grade only the final answer. And one LLM-judged benchmark, re-run 23 times on the same set-up, scored between 57.9% and 76.8% — an 18.9-point spread, bigger than the gaps between models that leaderboards report (Benchmarking the Benchmarks). A second audit reported in the EvalEval cost article found a "do-nothing" agent passing 38% of one airline benchmark's tasks under its original construction, and 12 of 17 agent benchmarks failing a basic held-out-set test (AI evals are becoming the new compute bottleneck).

The score can be measuring model size. A thesis fitted a statistical model to 4,395 models × 19 reasoning subtasks from the Open LLM Leaderboard and argues the two usual tools each miss half the problem: psychometric models treat "capability" as whatever the scores share (which ends up mostly meaning parameter count), and scaling laws assume the benchmark measures without error. Combining the two fitted the data better (RMSEA 0.0623 against 0.0876) (Quantifying construct validity). Its out-of-sample advantage is stated but was not quantified on fetch; do not cite a size for it.

Key Claims

• Tool-calling graders disagree with expert humans on 18.5% of tasks (92 of 496). Evidence: strong (preprint, expert review, four benchmark families) (Benchmarking the Benchmarks)
• The same set-up re-run 23 times spans 18.9 points on LiveMCPBench. Evidence: strong for that benchmark; not evidence about rule-graded benchmarks (same source)
• A do-nothing agent passed 38% of τ-bench airline tasks as originally built. Evidence: moderate (reported via HAL) (EvalEval)
• Combining latent-factor and scaling-law models fits leaderboard data better than either alone. Evidence: moderate (thesis, one leaderboard, BBH-shaped) (Quantifying construct validity)

Open Questions

• What share of a published leaderboard gap survives once grader error and repeat-run spread are subtracted? No source here has done that subtraction.
• Rule-graded and AI-judged benchmarks fail differently; nobody here reports repeat-run spread for a rule-graded one.

Related Concepts

• The Cost of Evaluation — fixing noise means repeat runs, and repeat runs are what make evaluation expensive.
• Agent Evaluation Benchmarks — the disclosure standard.

Changelog

• 2026-09-24 — Created: first compile of the two validity sources ingested 2026-09-11, plus the audit findings reported in the EvalEval cost article.