# The Binding Constraint Thesis

Canonical URL: https://menfem.com/kb/harnesses/concepts/binding-constraint-thesis
Knowledge base topic: [Harnesses & Agent Systems](https://menfem.com/kb/harnesses)
Frontier status: active
Tags: harness, agent-evaluation, methodology

---

For long-horizon agentic tasks across models of comparable capability, **the execution harness is often a stronger determinant of performance than the model**. The scaffolding — orchestration, tool wiring, retry and recovery policy, context management — decides how many tokens a finished job takes and whether it finishes at all.

The thesis carries a measured magnitude, which is what separates it from a plausible intuition. In a controlled **3×3 factorial** — GPT-5.4 / Kimi K2.6 / GLM-5.1 crossed with Minimal / Improved / Full harnesses, over a 100-task SWE-bench Verified subset at 2 runs per cell — variance **over harnesses** with the model fixed averaged **18.48 pp²**, against **2.37 pp²** over models with the harness fixed. That is the **7.80×** ratio, and it reproduced at run level (8.72× and 6.76×). More striking than the ratio: **6 of the 9 model-pair comparisons reversed their ranking** depending on which harness they ran under. Which model is "better" was decided by the scaffolding in two thirds of the pairings.

**Hold the coefficient carefully.** Per this KB's close read, the experiment is small (100 tasks, 2 runs) and **coding-only**, the accompanying leaderboard re-analysis inherits each source's uncontrolled conditions, and the piece is a **position paper** whose authors decline to treat 7.80× as a general constant. The direction and the ranking reversals are the robust findings; the coefficient is not portable to other task families.

The immediate methodological consequence is that **comparing two agents without disclosing their harnesses is a category error, not a stylistic omission**. A benchmark that reports "Model A beat Model B" while varying the scaffolding is measuring the scaffolding and attributing the result to the model. Most published agent comparisons do exactly this.

The economic consequence is why this rung exists at all. `models` measures capability per token and `inference-economics` measures dollars per token; the harness sits between them and sets how many tokens a *completed task* consumes, including retries and failed runs. At 7.80× variance, harness choice dominates model price differences of any realistic size — a cheaper model on a better harness routinely beats the reverse.

## Key Claims

- **Harness variance 18.48 pp² vs model variance 2.37 pp² — a 7.80× ratio**, with **ranking reversals in 6 of 9 model pairs**. Within-study, coding-only, not a general constant. *Evidence: moderate (controlled factorial, small n, position paper)* ([Stop Comparing LLM Agents Without Disclosing the Harness](../../raw/stop-comparing-llm-agents-disclosing-harness.md))
- **Agent comparisons that do not disclose the harness are not interpretable** as model comparisons. *Evidence: strong (paper)* ([Stop Comparing LLM Agents](../../raw/stop-comparing-llm-agents-disclosing-harness.md))
- **Substantial performance variation is attributable to harness configuration alone**, with task, budget, evaluation and model backend held fixed. *Evidence: strong (paper)* ([Harness-Bench](../../raw/harness-bench-measuring-harness-effects.md))

## Sources

- stop-comparing-llm-agents-disclosing-harness
- harness-bench-measuring-harness-effects

---

Cite as: MenFem Knowledge Base — https://menfem.com/kb/harnesses/concepts/binding-constraint-thesis