# Measuring Harness Effects

Canonical URL: https://menfem.com/kb/harnesses/concepts/harness-effect-measurement
Knowledge base topic: [Harnesses & Agent Systems](https://menfem.com/kb/harnesses)
Frontier status: active
Tags: harness, agent-evaluation, methodology

---

If the harness dominates the result, isolating it becomes the central experimental problem. The reference design is **Harness-Bench**: 106 sandboxed offline tasks and 5,194 execution trajectories, with **task, budget and evaluation held fixed while only the harness varies** across model backends.

**Fixing the budget is the load-bearing control**, and it is the one most often omitted. Without it, a "better harness" can simply be one that spends more inference — the improvement is real but it is bought, not designed, and it disappears the moment cost is held constant.

Harness-Bench's own headline finding names the dominant failure class: **execution-alignment decoupling** — the agent's reasoning drifting away from tool feedback, workspace state, or the output contract it was given. Cross-configuration variation shows up in completion, quality, efficiency *and* failure behaviour, not just pass rate.

That control produces the field's most useful negative result. **Automatic harness evolution — letting the system improve its own scaffolding — does not consistently outperform simple test-time-scaling baselines under matched feedback and inference budget, and what gains appear generalize poorly to held-out tasks** (measured on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6). Self-improving scaffolding is not yet a free lunch; much of its apparent advantage is unmatched spend.

Benchmarks themselves remain unsettled: a unified taxonomy covering roughly **60 agent benchmarks** exists, which is itself evidence that the field has not converged on what to measure.

## Key Claims

- **Harness-Bench: 106 tasks, 5,194 trajectories**, isolating configuration-level harness effects by fixing task/budget/eval. *Evidence: strong (paper)* ([Harness-Bench](../../raw/harness-bench-measuring-harness-effects.md))
- **Harness evolution does not beat test-time-scaling baselines under matched budget**, and generalizes poorly off-distribution. *Evidence: strong (paper)* ([Rethinking Harness Evolution](../../raw/rethinking-harness-evolution-evaluation.md))
- **~60 agent benchmarks catalogued** in a unified taxonomy — the field has not converged on a measurement standard. *Evidence: moderate (paper)* ([From LLM Reasoning to Autonomous Agents](../../raw/llm-reasoning-to-autonomous-agents.md))

## Sources

- harness-bench-measuring-harness-effects
- rethinking-harness-evolution-evaluation
- llm-reasoning-to-autonomous-agents

---

Cite as: MenFem Knowledge Base — https://menfem.com/kb/harnesses/concepts/harness-effect-measurement