Measuring Harness Effects

If the harness dominates the result, isolating it becomes the central experimental problem. The reference design is Harness-Bench: 106 sandboxed offline tasks and 5,194 execution trajectories, with task, budget and evaluation held fixed while only the harness varies across model backends.

Fixing the budget is the load-bearing control, and it is the one most often omitted. Without it, a "better harness" can simply be one that spends more inference — the improvement is real but it is bought, not designed, and it disappears the moment cost is held constant.

Harness-Bench's own headline finding names the dominant failure class: execution-alignment decoupling — the agent's reasoning drifting away from tool feedback, workspace state, or the output contract it was given. Cross-configuration variation shows up in completion, quality, efficiency and failure behaviour, not just pass rate.

That control produces the field's most useful negative result. Automatic harness evolution — letting the system improve its own scaffolding — does not consistently outperform simple test-time-scaling baselines under matched feedback and inference budget, and what gains appear generalize poorly to held-out tasks (measured on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6). Self-improving scaffolding is not yet a free lunch; much of its apparent advantage is unmatched spend.

Benchmarks themselves remain unsettled: a unified taxonomy covering roughly 60 agent benchmarks exists, which is itself evidence that the field has not converged on what to measure.

The rig now exists (2026-07-10)

Everything above describes measurement that had to be hand-built: Harness-Bench fixed task, budget and evaluation by construction, and the Rethinking audit had to assemble a matched-budget protocol from scratch. That is why so few of these comparisons exist — each one is a research project before it is a result.

Prime Intellect's verifiers release decomposes the environment into taskset / harness / runtime, with an interception server normalising OpenAI Chat Completions, OpenAI Responses and Anthropic Messages while recording traces identically across them. Any taskset runs under any compatible harness. Message-DAG traces cut storage from O(n²) to O(n) in turns, making 100+ turn rollouts practical — which is precisely the horizon where the failures on this rung live.

The consequence for this page: a matched-budget harness comparison stops being a bespoke build and becomes a configuration. The open question at the top of this rung — does a self-improving harness beat a plain one at equal spend — is now cheap to answer for anyone who wants to.

Two cautions before treating it as settled infrastructure. It is v0.2.0, a preview, despite universally being called "v1" in coverage. And the release note contains no benchmark — an interception proxy in the request path is exactly the sort of component that could cost something, and nothing here measures whether it does.

Source: verifiers v1 — primary vendor release note. Evidence: moderate for the architecture, none for performance.

Key Claims

• Harness-Bench: 106 tasks, 5,194 trajectories, isolating configuration-level harness effects by fixing task/budget/eval. Evidence: strong (paper) (Harness-Bench)
• Harness evolution does not beat test-time-scaling baselines under matched budget, and generalizes poorly off-distribution. Evidence: strong (paper) (Rethinking Harness Evolution)
• ~60 agent benchmarks catalogued in a unified taxonomy — the field has not converged on a measurement standard. Evidence: moderate (paper) (From LLM Reasoning to Autonomous Agents)