# Agent Evaluation Benchmarks

Canonical URL: https://menfem.com/kb/harnesses/concepts/agent-evaluation-benchmarks
Knowledge base topic: [Harnesses & Agent Systems](https://menfem.com/kb/harnesses)
Frontier status: active
Tags: evaluation, benchmarks, assessment

---

The evaluation landscape for LLM agents has matured rapidly, growing from isolated function-call correctness tests to comprehensive interactive benchmarks. Ferrag et al. consolidate approximately 60 benchmarks developed between 2019 and 2025 into a unified taxonomy spanning 8 domains: general knowledge reasoning, mathematical problem-solving, code generation, factual grounding, multimodal tasks, and interactive assessments.

Hu et al. propose a complementary three-tier evaluation framework specific to tool use: **tool-call validity** (does the model produce syntactically correct tool invocations?), **task completion** (does the tool chain achieve the goal?), and **interactive performance** (how well does the agent perform in realistic, multi-turn environments like WebArena and OSWorld?).

A critical finding across both surveys: benchmark performance does not reliably predict real-world deployment robustness. This gap represents one of the field's most pressing challenges.

## Longitudinal Capability Trends (AISI, Dec 2025)

Beyond static benchmark snapshots, the UK AI Security Institute's first **Frontier AI Trends Report** (Dec 2025) supplies the field's most rigorous *time-series* of frontier capability, drawn from controlled lab evaluations of **30+ models across 2022–2025**. The headline pattern is a measurable, accelerating doubling rate in security-relevant task horizons:

- **Cyber (Cybench-style):** apprentice-level task success rose from ~9–10% (late 2023) to ~50% (current); the *length* of cyber tasks models can complete unassisted is **doubling roughly every 8 months** — and recent (Feb 2026) internal data put that at **~4.7 months** since late 2024.
- **Software engineering:** ~one-hour SWE tasks succeeded **>40%** of the time (summer 2025).
- **Autonomy (RepliBench):** self-replication task success rose from **<5% (early 2023) to >60% (mid-2025)** across five frontier models — though confined to controlled, simplified environments.

These trends matter for evaluation because they show capability progress is best read as a *rate* (doubling time), not a fixed score — and that the rate is itself accelerating.

## The Matched-Budget Correction (Wang et al., Jul 2026)

The sharpest 2026 contribution to evaluation methodology is not a new benchmark but a **control**. Wang et al. (arXiv:2607.12227) show that a large class of reported agentic gains — those from **automatic harness evolution**, i.e. searching over scaffold/tool configurations — are confounded in two ways that generalize well beyond harness work:

1. **Budget confound.** Any method that iteratively evaluates and revises candidates using task feedback is spending inference compute. Unless it is compared against a simple task-level search baseline under **matched feedback and inference budget**, an improvement cannot be attributed to the method rather than to the extra compute it consumed.
2. **Search/eval benchmark sharing.** When the search loop and the reported evaluation use the same public benchmark, the headline number is partly a measure of overfitting to that task set. The corrective is a **held-out task split**.

Applied to Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6, harness evolution does not consistently beat simple test-time scaling and generalizes poorly to held-out tasks. Full treatment at [Agent Harness Evolution](./agent-harness-evolution.md).

The general lesson for this page: **matched budget and held-out evaluation should now be the default questions asked of any agentic result**, including vendor-published ones — which, as catalogued in [Inference & Token Economics](./inference-token-economics.md), essentially never disclose search or inference budget at all.

## Two more evaluation confounds the July pass measured: the harness, and the budget

The matched-budget correction has company. Three harness papers and one inference-compute paper (all ingested 2026-07-23) each name a *different* uncontrolled variable that current agent leaderboards hold implicit — and each shows it moving the number.

**The harness is an undisclosed variable that can dominate the model.** [Stop Comparing LLM Agents Without Disclosing the Harness](./agent-harness-evolution.md) measures **harness-induced variance at 7.80× model-induced variance** (18.48 vs 2.37 pp², 6/9 ranking reversals) in a controlled 3×3 SWE-bench factorial, and argues that long-horizon agent leaderboards are "incomplete and potentially misleading" until the harness is disclosed — proposing an **ETCSOVG Harness Card** and a variance-decomposition reporting protocol. [Harness-Bench](../../raw/harness-bench-measuring-harness-effects.md) supplies the diagnostic instrument (106 tasks, 5,194 trajectories) and shows performance must be attributed to *model-harness configurations*, not models. [Belief-divergence](../../raw/harness-induced-belief-divergence-multi-step-agents.md) shows that even *outcome-matched* harnesses drive different mid-trajectory beliefs — so pass/fail parity is not evidence of harness equivalence. Full treatment on [Agent Harness Evolution](./agent-harness-evolution.md); the evaluation lesson here is that **"which model scored X" is unanswerable without also stating the harness**, and no major leaderboard yet requires it.

**Inference budget is an evaluation *axis*, not a hidden constant.** [How Inference Compute Shapes Frontier LLM Evaluation](../../raw/how-inference-compute-shapes-frontier-llm-evaluation.md) (up to 12 frontier models × 7 benchmarks, controlled inference-scaling interventions: token-budget expansion, context compaction, repeated attempts) shows that **fixed single-budget scores increasingly *understate* models — and the underestimation grows for newer models**, which have a higher ceiling that only opens at generous budgets. Its recommendation is to report capability *as a function of* inference-time compute rather than as a point estimate. This is the other face of the matched-budget correction: Wang et al. warn that *unequal* budgets inflate a method's apparent gain; McFadyen et al. warn that a *fixed low* budget deflates a model's apparent ceiling. Together they make inference budget a first-class evaluation parameter — one that has to be swept and reported, not fixed and hidden.

The combined standard this page now carries: an agentic result is only interpretable if it discloses **(1) the harness, (2) the search/inference budget, and (3) held-out generalization** — and reports capability across a budget range, not at one point.

## Key Claims

- **~60 benchmarks exist across 8 evaluation domains** — Developed 2019-2025, covering reasoning, math, code, factual, multimodal, interactive. *Evidence: strong* ([From LLM Reasoning to Autonomous Agents](../../raw/llm-reasoning-to-autonomous-agents.md))
- **Three-tier evaluation framework for tool use** — Validity → completion → interactive performance. *Evidence: strong* ([Agentic Tool Use in LLMs](../../raw/agentic-tool-use-in-llms.md))
- **Benchmark-to-deployment gap persists** — Performance on benchmarks doesn't fully transfer to real-world scenarios. *Evidence: moderate* ([From LLM Reasoning to Autonomous Agents](../../raw/llm-reasoning-to-autonomous-agents.md))
- **Cyber-task time horizon is doubling every ~8 months, accelerating to ~4.7 months** — AISI's longitudinal evaluation of 30+ models (2022–2025); apprentice-level success 9%→50%. *Evidence: strong* ([AISI Frontier AI Trends Report](../../raw/aisi-frontier-ai-trends-report-2025.md))
- **Self-replication (RepliBench) success rose <5%→>60% (2023→mid-2025)** — but only in controlled, simplified settings; models are weakest at later stages (persistent access). *Evidence: strong* ([AISI Frontier AI Trends Report](../../raw/aisi-frontier-ai-trends-report-2025.md))
- **Gains from iterative search methods are unattributable without matched-budget baselines** — Harness evolution is structurally a test-time-scaling method; unless feedback and inference budget are equalized against a simple search baseline, "better method" and "more compute" cannot be separated. The argument is analytic and applies to any feedback-driven agentic method, not only harness search. *Evidence: strong* ([Rethinking the Evaluation of Harness Evolution](../../raw/rethinking-harness-evolution-evaluation.md))
- **Searching and reporting on the same benchmark inflates results; held-out splits are required** — On Terminal-Bench 2.1 (GPT-5.4, Claude Opus 4.6), evolved harnesses showed limited generalization to held-out tasks, indicating the search loop overfits the task set it optimizes against. *Evidence: strong* ([Rethinking the Evaluation of Harness Evolution](../../raw/rethinking-harness-evolution-evaluation.md))
- **The harness is an undisclosed evaluation variable that can dominate the model** — Harness-induced variance measured at 7.80× model-induced variance (18.48 vs 2.37 pp², 6/9 ranking reversals) on a controlled SWE-bench factorial; leaderboards without harness disclosure are "incomplete and potentially misleading." *Evidence: moderate (controlled factorial, small n, self-measured; position paper)* ([Stop Comparing](../../raw/stop-comparing-llm-agents-disclosing-harness.md), [Harness-Bench](../../raw/harness-bench-measuring-harness-effects.md))
- **Fixed single-budget scoring understates newer models; capability should be reported across an inference-budget range** — Up to 12 frontier models × 7 benchmarks under controlled inference-scaling interventions; underestimation grows for newer models with higher ceilings. Inference budget is an evaluation axis, not a hidden constant. *Evidence: moderate-to-strong (preprint, third-party, methodologically careful; numeric grid not captured)* ([How Inference Compute Shapes Frontier LLM Evaluation](../../raw/how-inference-compute-shapes-frontier-llm-evaluation.md))

## Benchmarks & Data

- ~60 benchmarks taxonomized (Ferrag et al.)
- Key interactive benchmarks: WebArena, OSWorld (Hu et al.)
- 11 real-world application sectors documented (Ferrag et al.)
- Terminal-Bench 2.1 used as the agentic testbed for the matched-budget correction (Wang et al.)
- Harness variance : model variance = **7.80×** on a 3×3 SWE-bench Verified factorial (Stop-Comparing); **106 tasks / 5,194 trajectories** isolating harness effects (Harness-Bench)
- Inference-budget sweep across up to **12 frontier models × 7 benchmarks** (SWE, math, medicine, cyber) shows fixed-budget scores understate newer models (McFadyen et al.)

## Open Questions

- How to build benchmarks that reliably predict real-world agent performance?
- Can evaluation keep pace with rapidly expanding agent capabilities?
- How to benchmark multi-agent collaborative scenarios fairly?
- What metrics capture safety failures, not just capability?
- How many previously published agentic gains survive a matched-budget, held-out re-evaluation? Nobody has swept the literature with this control yet.
- How should the matched-budget standard be enforced against vendor claims, where search and inference budgets are never disclosed?

## Related Concepts

- [Agentic Reasoning](./agentic-reasoning.md) — Benchmarks measure agentic capabilities
- [LLM Tool Use](./llm-tool-use.md) — Three-tier evaluation framework specific to tool use
- [Multi-Agent Systems](./multi-agent-systems.md) — Interactive benchmarks increasingly test collaboration
- [Agent Safety & Alignment](./agent-safety-alignment.md) — The AISI report's biorisk/jailbreak findings sit on the safety side
- [Agent Harness Evolution](./agent-harness-evolution.md) — the specific literature the matched-budget correction was aimed at; full home of the 7.80× harness-variance finding and the harness-disclosure cluster
- [Inference & Token Economics](./inference-token-economics.md) — where "capability is a function of inference budget" becomes a per-task pricing argument

## Backlinks

*Pages that reference this concept:*
- [OpenAI](../entities/openai.md)
- [DeepSeek](../entities/deepseek.md)
- [Agent Harness Evolution](./agent-harness-evolution.md)

## Changelog

- **2026-06-24** — Compiled new source (aisi-frontier-ai-trends-report-2025): added longitudinal capability-trend section (Cybench doubling ~8mo→~4.7mo, RepliBench <5%→>60%, 1hr SWE >40%)
- **2026-07-22** — Compiled new source (rethinking-harness-evolution-evaluation): added the matched-budget / held-out-split correction as a general evaluation standard, two claims, and open questions on re-evaluating prior results and vendor claims; frontier_status steady → active
- **2026-07-23** — Compiled four evaluation-methodology sources. Added a "Two more evaluation confounds" section: the **harness** as an undisclosed variable (Stop-Comparing 7.80×, Harness-Bench 5,194 trajectories, belief-divergence) and **inference budget** as a first-class evaluation axis (How Inference Compute Shapes Frontier LLM Evaluation — fixed-budget scoring understates newer models). Consolidated the emerging standard: disclose harness + budget + held-out, report capability across a budget range. Two new key claims and two benchmark-data rows. Deep treatments stay on Agent Harness Evolution and Inference & Token Economics.

## Sources

- llm-reasoning-to-autonomous-agents
- agentic-tool-use-in-llms
- aisi-frontier-ai-trends-report-2025
- rethinking-harness-evolution-evaluation
- stop-comparing-llm-agents-disclosing-harness
- harness-bench-measuring-harness-effects
- harness-induced-belief-divergence-multi-step-agents
- how-inference-compute-shapes-frontier-llm-evaluation

---

Cite as: MenFem Knowledge Base — https://menfem.com/kb/harnesses/concepts/agent-evaluation-benchmarks