Rethinking the Evaluation of Harness Evolution for Agents

Abstract

"We revisit the evaluation of automatic harness evolution for LLM agents. Existing harness evolution methods use unit test cases to search for harness configurations and then report final performance on the same public benchmark. This protocol raises two fundamental concerns. First, harness evolution is itself an iterative search procedure that repeatedly evaluates and revises candidate harnesses using task feedback. As in agentic test-time scaling, it should therefore be compared with simple task-level search baselines under matched feedback and inference budgets to determine whether its gains arise from improved harness design or from additional search alone. Second, because the search and the final evaluation share the same benchmark, the reported gains risk overfitting to that specific task set. To address these concerns, we conduct an extensive evaluation comparing harness evolution with simple test-time scaling and discovery baselines under comparable feedback and inference budgets, and also evaluate evolved harnesses on held-out tasks to assess whether the discovered improvements generalize. Experiments on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6 show that automatic harness evolution does not consistently outperform simple test-time scaling methods and exhibits limited generalization."

Key Contributions

• Names and formalizes two evaluation flaws in the harness-evolution literature: (1) conflating "better harness design" with "more search/inference budget spent," and (2) train/test leakage from evaluating evolved harnesses on the same benchmark used to search for them.
• Proposes a matched-budget protocol: compare harness evolution against simple test-time-scaling and discovery baselines under equal feedback and inference budget.
• Adds a held-out generalization test: evaluate the evolved harness on tasks it never saw during search.
• Empirically demonstrates on Terminal-Bench 2.1 (using GPT-5.4 and Claude Opus 4.6 as the underlying models) that harness evolution's reported gains largely do not survive matched-budget comparison or hold-out generalization.

Methodology

• Terminal-Bench 2.1 as the agentic benchmark.
• Two backbone models: GPT-5.4, Claude Opus 4.6.
• Three comparison arms: (1) automatic harness evolution, (2) simple test-time scaling, (3) discovery baselines — all under matched feedback/inference budget.
• Held-out task split to test whether harness improvements transfer beyond the search-time benchmark.

Results

• Automatic harness evolution does not consistently outperform simple test-time scaling under equal budget — meaning published harness-evolution gains may be attributable to spending more inference/search compute, not to a genuinely better tool/scaffold design.
• Evolved harnesses show limited generalization to held-out tasks — evidence of benchmark-specific overfitting in the harness-search loop itself.

Limitations

• Institution not explicitly stated in the fetched abstract; affiliations of Hajishirzi/Tsvetkov (UW) and Dasigi (AI2) are well known externally but not confirmed from this source alone — flagged as inferred, not sourced.
• Scope limited to Terminal-Bench 2.1 with two specific backbone models; whether the finding generalizes to other agentic benchmarks (SWE-bench, GAIA) is untested here.
• Full paper body not ingested — abstract-level fetch only (summary-derived).

Why this matters for the KB

Directly serves the content-focus.md "harness-agentic layer" lane and the AI LENS's "Agents & tool use" required chapter. It's a methodological check on the entire "agent scaffolding = free capability" narrative MenFem has been tracking (frontier.md's Agent Evaluation Standardization + Agent Memory Architectures frontiers) — a caution flag for any future MenFem coverage of harness/scaffold vendors claiming benchmark gains from evolved tool configs rather than raw model capability or spent compute.

Source: Rethinking the Evaluation of Harness Evolution for Agents (arXiv:2607.12227), Wang et al., submitted 2026-07-14.