Evaluation & Measurement — Frontier

The open questions on the rung that decides whether any other rung's number can be believed. Compiled 2026-08-28 from four close-reads; recompiled 2026-09-24 with three more (the eval-cost article and the two validity sources ingested 2026-09-11). This is still a thin rung and the compile does not disguise it — see the coverage note at the end.

The binding constraint, named

A benchmark score without a stated compute budget is not a measurement.

The evidence is direct: across up to 12 frontier models and 7 benchmarks (software engineering, maths, medicine, cybersecurity) under controlled inference-scaling interventions — larger token budgets, context compaction, repeated attempts — fixed single-budget evaluations increasingly understate newer models, because newer models have a higher ceiling at generous budgets (How Inference Compute Shapes Frontier LLM Evaluation). The paper's recommendation is the constraint stated positively: report capability as a function of inference-time compute, not as a scalar.

That finding is why this rung exists on an inference-stack Atlas at all. If capability is a function of budget, then score and cost are the same axis — and a leaderboard that publishes one without the other cannot be converted into a task price. Every rung downstream that wants to turn a token price into a task price depends on this one refusing scalar scores.

The number this rung owns

It does not have one yet, and that is the honest answer.

The closest candidate is the 12-model × 7-benchmark inference-scaling grid (as-of 2026-06-16), but the close-read explicitly records that the per-benchmark numeric grid was not captured — only the qualitative direction. So this rung currently owns a shape (capability rises with budget, and the gap widens for newer models) without owning a number.

Closing that is the single most valuable ingest available here.

A second number, on the cost side

The rung now holds one hard figure of its own kind: what a score costs to produce. About $40,000 for one agent leaderboard run once per cell (9 models × 9 benchmarks, 21,730 runs), roughly $320,000 if each cell is repeated 8 times for a trustworthy number, and a 33× cost spread on identical tasks from the agent set-up alone — as of 2026-04-29 (AI evals are becoming the new compute bottleneck). These are compiled from other sources by an advocacy group, not measured by it; cite them as reported. See the cost of evaluation.

Active questions

Question	State of evidence	What would settle it
How much does a score move across the budget sweep?	Direction established, magnitude not held here. The one source with the grid did not have it extracted. Externally, the picture is worse than "uncertain" — the same model has been reported moving 73% → 89% while cost moved $0.91 → $2.34, with no model change at all (Rails Foundation study, held on the research sheet, not yet ingested to this rung).	Capturing the per-benchmark grid from the existing source, then one owned replication.
Are benchmark versions comparable at all?	Actively not. Terminal-Bench 3.0 compresses scores roughly 2.5x against 2.1 on the same task family, and both versions circulate in vendor material without the version prominently attached. (Research sheet, 2026-08-20 — not yet an ingested source here.)	A version-to-version calibration table. Nobody publishes one.
Does a frontier capability trend survive independent testing?	Partly. AISI's is the only multi-year independent series here — 30+ models tested 2022–2025 across cyber, biology, chemistry, autonomy/self-replication, software engineering and safeguards. It is the rung's most credible instrument precisely because the tester owns no model.	Continuation of the series, and any second national institute publishing a comparable one.
What does a trustworthy score cost?	Now priced, roughly. A single run of one LLM-judged benchmark is not a reading — 23 repeats of one set-up spanned 57.9%–76.8% (Benchmarking the Benchmarks) — and repeating every cell 8× takes one leaderboard from ~$40,000 to ~$320,000. Agent benchmarks compress only 2–3.5×, against 100–200× for static ones.	Repeat-run spreads published per benchmark, so a buyer knows how many runs each needs before its gaps mean anything.
How often is the grader wrong?	Measured once: 18.5% grader–expert disagreement over 496 tool-calling tasks. A method for asking whether a score measures the named ability or model size exists (4,395-model fit), but its out-of-sample gain was not quantified on fetch (Quantifying construct validity).	The same audit on a rule-graded coding benchmark.
Is contamination or eval-set overfitting measurable in the open?	Unknown — the rung holds nothing on it, despite it being named in scope in LENS.md.	Any source at all.
Does matched-budget control change published rankings?	Open and high-stakes. The harnesses rung already records that automatic harness evolution does not beat simple test-time scaling under matched budget — the same control, applied to model leaderboards, has not been done publicly.	A leaderboard re-run under a single fixed budget.

Coverage note — read this before citing the rung

Seven sources, and two of them are off the rung's through-line. PhyWorldBench (12,600 videos, physical realism in text-to-video) and VideoScience-Bench (Sora-2, Veo-3 and peers reaching only ~58–64% on Phenomenon Congruency) are genuine benchmark-design work, but they measure video generation physics, not the inference stack this Atlas exists to price. They were ingested 2026-04-22, before the rung was rescoped.

So the working count for this rung's actual purpose is five sources. Of those, the cost figures come from one blog post compiling other people's numbers, and the construct-validity source is a thesis.

Two things in scope still have zero coverage: contamination and overfitting to the eval set, and the eval-to-deployment gap.

Standing note

Our seat here is read it. No evaluation in this rung has been run by us.

This is the rung where that matters most, because its whole claim is that other people's numbers should not be taken at face value — and a rung making that argument on seven sources, two off-topic, and no owned number is making it on borrowed authority. Deep-dive menu #2 (the frozen-harness benchmark on his own corpus) would convert this rung and the serving rung in the same run.