Stop Comparing LLM Agents Without Disclosing the Harness

Abstract

Verbatim (opening): "This position paper argues that, for long-horizon tasks evaluated across models with comparable frontier capability, the agent execution harness, namely the infrastructure layer that governs context construction, tool interaction, orchestration, and verification around a language model, is often a stronger determinant of agent performance than the model it wraps."

The paper formalizes this as the Binding Constraint Thesis: in the comparable-capability regime, performance variance is governed more by harness configuration than by model choice, so current evaluation protocols systematically misattribute harness-level gains to model improvements. Until harness specifications are disclosed, the paper argues, leaderboard comparisons for long-horizon agents should be treated as incomplete and potentially misleading.

Key Contributions

• The Binding Constraint Thesis, supported three ways: (1) a control-theoretic model treating the harness as a closed-loop controller around the model; (2) empirical evidence that harness-induced variance can exceed model-induced variance, including model ranking reversals; (3) a proposed harness-aware evaluation framework (disclosure + variance decomposition).
• ETCSOVG Harness Card — a seven-layer disclosure taxonomy required in submission cards: Execution (runtime, sandboxing, timeout, step budget), Tool (tool list, schema style, error format, retryable classes), Context (window cap, compression policy, retrieval, memory), Scheduling (agent loop, retry, escalation, rollback), Observability (logged artifacts, trajectory format, checkpoints), Verification (parser, validation, self-checking, anomaly detection), Governance (permission model, allow/denylist, side-effect boundaries).
• Variance-decomposition protocol — for any model-by-harness grid, report: HV per model (variance over harnesses, model fixed), MV per harness (variance over models, harness fixed), the aggregate ratio HV̄/MV̄, the count of model-pair ranking reversals, and a partial eta-squared interaction effect size.
• Trajectory-level metrics — Recovery Rate RR(k) (probability of moving from an anomaly state to a task-advancing state within k steps), Context Retention (fraction of task-relevant files/tests/constraints present in constructed context per step), and Control Lag τ (steps between harness-detected anomaly and corrective signal reaching the policy).

Methodology

Controlled factorial experiment (Section 4.2): three frontier models clustered near ~45% on a coding leaderboard — GPT-5.4, Kimi K2.6, GLM-5.1 — crossed with three harness configurations on a difficulty-stratified 100-task SWE-bench Verified subset; two independent runs per cell, shared task order, Docker env, 50-step budget, 120s timeouts.

• H₁ (Minimal): no context compression, verbose schemas, no retry, no verification.
• H₂ (Improved): compressed context with retrieval, structured retry with backoff, basic feedback.
• H₃ (Full): H₂ + per-step self-checking, KL-drift checks, anomaly detection, rollback.

Plus a re-analysis of public leaderboards (SWE-bench Pro, SWE-bench Verified Mini / HAL, Terminal-Bench) where the same model appears under different scaffolds.

Results

Controlled experiment (Table 2):

• GLM-5.1: 52.5% (H₁) → 56.5% (H₂) → 65.5% (H₃); harness variance 29.56 pp².
• GPT-5.4: 55.0% → 58.5% → 63.5%; HV 12.17 pp².
• Kimi K2.6: 52.0% → 59.0% → 60.5%; HV 13.72 pp².
• Average HV 18.48 pp² vs average MV 2.37 pp² — a 7.80× ratio. Harness changes moved GLM-5.1 by 13.0 pp; model swaps within a fixed harness moved results only 3.0–5.0 pp.
• Ranking reversals in 6 of 9 model-pair / harness-pair comparisons.
• Run-level check (Table 7): HV/MV = 8.72× (first run) and 6.76× (second) — "not an artifact of averaging."

Public-leaderboard evidence (Section 4.1 / Table 1):

• SWE-bench Pro, Claude Opus 4.5: SEAL scaffold 45.9% → Claude Code harness 55.4% (+9.5 pp); within-model harness range ≈ 2× the within-harness model range. Adding a WarpGrep search subagent (+2.1–2.2 pp) flipped the MiniMax 2.5 vs Claude Opus 4.6 ordering.
• SWE-bench Verified Mini (HAL): Claude Sonnet 4.5 shows a 34-pt scaffold gap (68% vs 34%); GPT-5 Medium 34-pt gap (46% vs 12%); o4-mini a 48-pt gap.
• Third-party monitoring: up to 11–15 pp of scaffold-only variation on SWE-bench Verified; "scaffold choice has the single biggest impact on overall performance."
• APEX-Agents (480 professional tasks, Appendix D): agent failures were "predominantly execution and orchestration problems rather than knowledge failures"; zero-score rates varied 40–62% using the same underlying model.

Limitations

The controlled experiment is small (3×3, 100 tasks, 2 runs) and coding-only; the leaderboard re-analysis inherits each source's uncontrolled conditions. As a position paper the central claim is argued rather than proven at scale, and the 7.80× ratio is a within-study measurement, not a general constant. The disclosure standard is a proposal, not yet adopted by any leaderboard.

Source: arXiv:2605.23950 — Stop Comparing LLM Agents Without Disclosing the Harness, Zhang, Wang, Ge, Xu, Hamm, Reddy, 7 May 2026. Abstract + full HTML5 retrieved 2026-07-23.