Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

Abstract

Verbatim: "LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete agent systems, or hold the harness fixed, making execution-layer variation difficult to study. We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent workflows."

Key Contributions

• Isolates the harness as the experimental variable. Where prior benchmarks compare whole agent systems or freeze the harness, Harness-Bench evaluates representative harness configurations across multiple model backends under shared task environments, budgets, and evaluation protocols — so execution-layer variation becomes measurable rather than confounded.
• A realistic, execution-grounded task suite: 106 sandboxed offline tasks derived from practical agent patterns (tool use, workspace modification, concrete artifact production).
• Scale of evidence: 5,194 execution trajectories across model-harness pairings.
• Names a dominant failure class: "recurring execution-alignment failures, where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts" — i.e., failures live in the execution loop, not in the model's knowledge.

Methodology

Configuration-level diagnostic design: fix the task environment, the compute/step budget, and the evaluation protocol; vary the harness configuration across a set of model backends; measure completion, process quality, efficiency, and failure behavior per model-harness cell. The 106 tasks are offline and sandboxed so runs are reproducible and the execution layer is fully observed.

Results

• "Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior across model-harness pairings."
• The headline conclusion is attributional: agent performance should be reported for model-harness configurations, not models in isolation.
• The recurring failure mode is execution-alignment decoupling — reasoning that reads as plausible but has drifted from tool feedback / workspace state / output contracts.

Limitations

Tasks are offline and sandboxed (not live production traffic), which trades ecological validity for reproducibility. The paper characterizes variation and failure classes but (from the abstract-level read) does not by itself resolve which harness features cause which gains — it is a measurement instrument. The specific model backends are not enumerated on the abstract page.

Relation to the KB

Direct empirical complement to Stop Comparing LLM Agents Without Disclosing the Harness: where that paper argues the Binding Constraint Thesis and re-analyzes leaderboards, Harness-Bench builds the controlled instrument to measure the same effect. Both sit under frontier §5 (Harness Layer) and §12 (Agent Evaluation Standardization), and both bear on the MenFem thesis that the edge is in the harness, not the model.

Source: arXiv:2605.27922 — Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows, Yao, Tan, Liu, Li, Wang, Yu, Tan, Tian, Zhao, Sun, Zhang, Yang, 27 May 2026. Abstract page retrieved 2026-07-23.