The Binding Constraint Thesis

For long-horizon agentic tasks across models of comparable capability, the execution harness is often a stronger determinant of performance than the model. The scaffolding — orchestration, tool wiring, retry and recovery policy, context management — decides how many tokens a finished job takes and whether it finishes at all.

The thesis carries a measured magnitude, which is what separates it from a plausible intuition: harness-induced variance was measured at 7.80× model-induced variance — swapping the harness moved the result almost eight times as much as swapping the model.

Hold that number carefully. The source's own limits, per this KB's close read: the controlled experiment is small (3×3, 100 tasks, 2 runs) and coding-only, the accompanying leaderboard re-analysis inherits each source's uncontrolled conditions, and the piece is a position paper whose central claim is argued rather than proven at scale. 7.80× is a within-study measurement, not a general constant. The direction of the finding is well supported; the coefficient is not yet portable to other task families.

The immediate methodological consequence is that comparing two agents without disclosing their harnesses is a category error, not a stylistic omission. A benchmark that reports "Model A beat Model B" while varying the scaffolding is measuring the scaffolding and attributing the result to the model. Most published agent comparisons do exactly this.

The economic consequence is why this rung exists at all. models measures capability per token and inference-economics measures dollars per token; the harness sits between them and sets how many tokens a completed task consumes, including retries and failed runs. At 7.80× variance, harness choice dominates model price differences of any realistic size — a cheaper model on a better harness routinely beats the reverse.

Key Claims

• Harness-induced variance measured at 7.80× model-induced variance on coding tasks — a within-study figure from a 3×3, 100-task controlled experiment, not a general constant. Evidence: moderate (position paper, small controlled study) (Stop Comparing LLM Agents Without Disclosing the Harness)
• Agent comparisons that do not disclose the harness are not interpretable as model comparisons. Evidence: strong (paper) (Stop Comparing LLM Agents)
• Substantial performance variation is attributable to harness configuration alone, with task, budget, evaluation and model backend held fixed. Evidence: strong (paper) (Harness-Bench)