# Agent Harness Evolution

Canonical URL: https://menfem.com/kb/ai/concepts/agent-harness-evolution
Knowledge base topic: [Artificial Intelligence](https://menfem.com/kb/ai)
Frontier status: active
Tags: agent-harness, evaluation-methodology, test-time-scaling, agentic-layer

---

The **harness** is everything wrapped around a model to make it an agent: the tool definitions, the system prompt, the memory store, the retry and verification logic, the orchestration loop. **Harness evolution** is the automated version of designing it — an outer search loop that proposes candidate harness configurations, evaluates each against task feedback, and keeps the winners. It is the mechanism behind a widespread claim: that large agentic gains are available from scaffolding alone, on top of a fixed model.

Two distinct questions live on this page, and 2026 answered them in opposite directions. First: **does the harness layer determine agent performance more than the model does?** The July discovery pass ingested three papers that measure this directly, and the answer is a firm yes — with harness-induced variance running **7.80× model-induced variance** in a controlled experiment. Second: **can that harness advantage be *discovered automatically* by searching over configurations, and does it transfer?** Wang et al.'s matched-budget audit says no, or at least not yet. The two findings are consistent — the harness is where outcomes are decided; *automating its design by benchmark search* is the part that failed — but they must be kept separate, and this page had previously carried only the second.

## The harness effect, measured directly (May–Jul 2026)

Three papers, all ingested 2026-07-23, move "the harness matters" from assertion to measurement. They matter *more* than the harness-evolution audit below, because they pit the harness layer against the model layer head-to-head rather than one harness technique against another.

**Stop Comparing LLM Agents Without Disclosing the Harness** (Zhang et al., arXiv:2605.23950, May 2026) formalizes the **Binding Constraint Thesis**: for long-horizon agentic tasks across models of comparable frontier capability, the execution harness is often a stronger determinant of performance than the model it wraps. In a controlled 3×3 factorial — three models clustered near ~45% on a coding leaderboard (GPT-5.4, Kimi K2.6, GLM-5.1) crossed with three human-configured harnesses (Minimal / Improved / Full), on a difficulty-stratified 100-task SWE-bench Verified subset, two runs per cell — **harness-induced variance was 7.80× model-induced variance: 18.48 vs 2.37 pp², with model-pair ranking reversals in 6 of 9 comparisons.** Swapping the harness moved GLM-5.1 by 13.0 pp (52.5% → 65.5%); swapping the model inside a fixed harness moved results only 3.0–5.0 pp. Run-level ratios of 8.72× and 6.76× rule out an averaging artifact. Public-leaderboard re-analysis corroborates the direction (Claude Opus 4.5 +9.5 pp from SEAL → Claude Code scaffold; 34–48-pt scaffold gaps for the same model on HAL's SWE-bench Verified Mini). **This is the "edge is the harness, not the model" thesis measured in a controlled experiment — the single most direct evidentiary support for the house thesis anywhere in this KB.** The paper also proposes an **ETCSOVG Harness Card** disclosure standard (seven layers: Execution, Tool, Context, Scheduling, Observability, Verification, Governance) plus a variance-decomposition reporting protocol (per-model HV, per-harness MV, the ratio HV̄/MV̄, ranking-reversal count, partial η²).

**Harness-Bench** (Yao et al., arXiv:2605.27922, May 2026) is the empirical instrument for the same effect: **106 sandboxed offline tasks, 5,194 execution trajectories**, holding task/budget/eval fixed and varying only the harness across model backends. It reports substantial variation in completion, process quality, efficiency, and failure behavior across model-harness pairings, and names **execution-alignment decoupling** — plausible reasoning drifting from tool feedback, workspace state, evidence, or output contracts — as the dominant failure class. Its headline conclusion is attributional: report performance for *model-harness configurations*, not models in isolation.

**Harness-induced belief divergence** (Yi & Song, arXiv:2607.04528, Jul 2026) goes one level deeper: the harness changes an agent's *multi-step beliefs* — progress, risk, recoverability, constraints, failure mode, uncertainty, future success, repair cost, next action — **even when task, environment, model, and terminal success are all held fixed.** Common harness moves (blocked actions, compressed repairs, selective verification, cost-aware evidence pruning) often preserve pass/fail while altering the beliefs that drive later decisions. A belief-rollout diagnostic decomposes cross-harness divergence into an *arrival* term (immediate interface shift) and a *growth* term (horizon-dependent drift); BIWM is a no-training protocol that aligns belief trajectories across harness views. The upshot: two harnesses that score identically are still not interchangeable — outcome parity hides mid-trajectory divergence.

**Read together:** Stop-Comparing measures the harness's effect on terminal *performance* (7.80×), Harness-Bench on *outcomes and failure behavior* (5,194 trajectories), and belief-divergence on the agent's *internal beliefs* (matched success, divergent beliefs). All three converge on one claim — the harness is an experimental variable, not an implementation detail — and all three study *human-configured* harnesses, which is precisely the arm the harness-evolution audit below does not test.

Wang et al. (arXiv:2607.12227, July 2026) subject that second claim — automatic discovery — to a methodological audit, and the result is a caution flag rather than a refutation. They identify **two flaws in how harness evolution has been evaluated**:

1. **The budget confound.** Harness evolution is *itself* an iterative search that repeatedly evaluates and revises candidates using task feedback — structurally the same thing as agentic test-time scaling. If it is not compared against simple task-level search under **matched feedback and inference budget**, there is no way to tell whether the reported gains come from a better harness design or merely from having spent more search compute.
2. **The shared-benchmark leak.** Because the search loop and the final evaluation use the *same* public benchmark, reported gains risk being overfitting to that specific task set rather than genuine improvement.

Their corrective protocol is to run harness evolution against simple test-time-scaling and discovery baselines under comparable budgets, and to additionally evaluate evolved harnesses on **held-out tasks**. On Terminal-Bench 2.1, with GPT-5.4 and Claude Opus 4.6 as backbones, automatic harness evolution **does not consistently outperform simple test-time scaling** and **exhibits limited generalization**.

## What this does and does not say

This distinction is load-bearing and easy to get wrong in both directions.

**What the paper establishes:**
- *Automatically evolved* harnesses, scored under matched budget, do not reliably beat just spending that same budget on task-level search.
- Gains discovered by searching on a benchmark do not transfer well off that benchmark.
- Published harness-evolution results should be discounted unless they report matched-budget baselines and held-out evaluation.

**What the paper does not establish:**
- It does **not** compare "good harness" against "no harness." There is no arm here in which an agent runs without scaffolding.
- It does **not** evaluate human-designed harnesses. The target is the automated search procedure, not the artifact.
- It does **not** claim harness design is irrelevant — only that *automating the design by benchmark search* has not been shown to add value beyond the compute it consumes.
- Scope is two backbone models on one benchmark (Terminal-Bench 2.1). Whether it replicates on SWE-bench, GAIA, or OSWorld is untested here.

## Bearing on the "edge is the harness layer, not the model" thesis

**It complicates the thesis without overturning it, and it raises the evidentiary bar for anyone asserting it.**

*Cutting against:* the specific mechanism by which harness advantage was supposed to become cheap and compounding — automated search over scaffolds, run against a benchmark, yielding transferable gains — does not survive contact with matched-budget controls. If harness advantage cannot be discovered automatically and does not generalize off the task set it was found on, then it is not a scalable, self-improving moat. It looks more like bespoke engineering that has to be re-earned per domain.

*Consistent with the thesis:* the paper's own finding is that **test-time scaling — spending more inference compute inside a loop — matches the evolved harness**. Test-time search *is* harness-layer work. What is being demoted is one particular harness technique relative to another, not the layer relative to the model. Both arms hold the model fixed. Nothing here suggests the gains came from the weights.

*Corroborating evidence already in this KB, from a different angle:* the Amazing Agent Race benchmark found Claude Code matching Codex CLI at 37% accuracy while using **6× fewer tokens** — a large, real, harness-attributable difference in cost at equal capability, between two human-designed harnesses on the same underlying task. That is harness-layer edge measured directly, and it is an efficiency edge rather than a capability edge. See [Tool-Chain Navigation](./tool-chain-navigation.md).

*The honest synthesis:* the harness layer is where cost, reliability, *and a large share of measured performance* are determined — and the evidence for that is now direct, not merely inferred. Stop-Comparing's 7.80× (harness variance : model variance) puts a number on it under controlled conditions; Harness-Bench and belief-divergence corroborate it on outcomes and on internal beliefs. What did *not* survive is the narrower, more valuable claim that harness advantage can be *discovered automatically by benchmark search and transfers* — that is the part that just failed its first serious audit. So the thesis splits cleanly: "the harness beats the model" is measured and standing; "the harness edge is a cheap, self-improving, transferable moat" is unproven and, for automated search specifically, contradicted. MenFem coverage should treat any vendor claiming benchmark improvements from an evolved tool configuration as unproven until they show matched-budget baselines and held-out tasks — the same standard that should be applied to the vendor token-efficiency claims catalogued in [Inference & Token Economics](./inference-token-economics.md) — while treating the *general* "harness > model" claim as now empirically grounded.

## Meanwhile, the harness is shipping as commercial infrastructure

While the research result is a caution, the **productization is accelerating** — and the primary-source record for that is unusually good, because it is a first-party changelog rather than a marketing page. Anthropic's Claude Platform release notes for June 25 – July 15, 2026 show the harness being standardized into billable API surface:

- A new **`agent-memory-2026-07-22` beta header** replaces `managed-agents-2026-04-01` on memory-store endpoints, tightening listing semantics (server-defined stable ordering, `depth` restricted to `0`/`1`/omitted, `path_prefix` matching whole path segments and required to end in `/`). Sending both headers is a 400. All major SDKs now send the new header by default; the old behaviour retires 2026-07-22.
- **Managed Agents session streams** gained opt-in event deltas, backward pagination, and **per-session config override** — a session can swap model, system prompt, tools, MCP servers, and skills without touching the base agent. That is the harness becoming a runtime-configurable object.
- **Dreams**, Anthropic's newer managed-agent surface, expanded to Claude Fable 5 and Claude Sonnet 5 (July 10, research preview).

The gap between these two facts is itself the finding: the harness is being sold as infrastructure faster than the evidence that automating its design produces transferable gains has been established.

## Key Claims

- **Harness-induced performance variance is 7.80× model-induced variance** — In a controlled 3×3 factorial (GPT-5.4 / Kimi K2.6 / GLM-5.1 × Minimal / Improved / Full harness, 100-task SWE-bench Verified subset, 2 runs/cell), variance over harnesses (model fixed) averaged 18.48 pp² vs 2.37 pp² over models (harness fixed) — a **7.80× ratio**, with ranking reversals in 6 of 9 model-pair comparisons and run-level ratios of 8.72× / 6.76×. The direct measurement of the "edge is the harness, not the model" thesis. *Evidence: moderate — a controlled factorial experiment (which weighs more than any vendor self-report), but small n (100 tasks, 2 runs, coding-only), self-measured, a within-study ratio the authors decline to treat as a general constant; position paper, not peer-reviewed* ([Stop Comparing LLM Agents Without Disclosing the Harness](../../raw/stop-comparing-llm-agents-disclosing-harness.md))
- **Agent performance must be attributed to model-harness *configurations*, not models alone** — Harness-Bench isolates the harness as the experimental variable across 106 tasks / 5,194 trajectories and finds substantial cross-configuration variation in completion, quality, efficiency, and failure behavior; execution-alignment decoupling (reasoning drifting from tool feedback / workspace state / output contracts) is the dominant failure class. *Evidence: moderate (diagnostic benchmark; offline/sandboxed for reproducibility, not live traffic)* ([Harness-Bench](../../raw/harness-bench-measuring-harness-effects.md))
- **The harness alters an agent's multi-step beliefs even at matched terminal success** — Blocked actions, compressed repairs, selective verification, and cost-aware evidence pruning preserve pass/fail while changing the beliefs (progress, risk, recoverability, next action, …) that drive later decisions; divergence splits into an arrival (interface) term and a growth (horizon) term. Outcome parity is not harness equivalence. *Evidence: weak-to-moderate (preprint, 2 authors, no headline magnitude in the abstract-grounded ingest)* ([Harness-Induced Belief Divergence](../../raw/harness-induced-belief-divergence-multi-step-agents.md))
- **Harness disclosure is not yet a leaderboard norm** — Stop-Comparing argues long-horizon agent leaderboards are "incomplete and potentially misleading" until harness specs are disclosed, and proposes the ETCSOVG Harness Card + variance-decomposition protocol as the fix. It is a proposal, adopted by no leaderboard as of ingest. *Evidence: moderate (the gap is real and documented; the standard is aspirational)* ([Stop Comparing](../../raw/stop-comparing-llm-agents-disclosing-harness.md))
- **Automatic harness evolution does not consistently outperform simple test-time scaling under matched feedback and inference budget** — Terminal-Bench 2.1, backbones GPT-5.4 and Claude Opus 4.6, compared against test-time-scaling and discovery baselines. *Evidence: strong (paper with an explicitly controlled protocol; the whole contribution is the control)* ([Rethinking the Evaluation of Harness Evolution](../../raw/rethinking-harness-evolution-evaluation.md))
- **Evolved harnesses generalize poorly to held-out tasks** — Because the search and the final evaluation share the same benchmark, reported gains risk being task-set overfitting. Held-out evaluation shows limited transfer. *Evidence: strong (direct experimental test of the specific failure mode)* ([Rethinking the Evaluation of Harness Evolution](../../raw/rethinking-harness-evolution-evaluation.md))
- **Harness evolution is structurally a test-time-scaling method and must be budgeted as one** — It repeatedly evaluates and revises candidates using task feedback; without a matched-budget comparison, "better harness" and "more search" are not separable. This is a methodological claim that applies beyond the two models tested. *Evidence: strong (argument is analytic, not empirical — it holds independent of the experiment)* ([Rethinking the Evaluation of Harness Evolution](../../raw/rethinking-harness-evolution-evaluation.md))
- **The scope is narrow: one benchmark, two backbones, automated search only** — No arm tests human-designed harnesses or harness-vs-no-harness. *Evidence: strong (stated limitation)* ([Rethinking the Evaluation of Harness Evolution](../../raw/rethinking-harness-evolution-evaluation.md))
- **Agent memory and session configuration are now versioned, billable platform primitives** — Anthropic shipped a breaking beta-header migration on memory-store endpoints and per-session harness override on Managed Agents inside a three-week window. *Evidence: strong for what shipped (first-party changelog); this source says nothing about how well any of it works* ([Anthropic Platform Release Notes](../../raw/anthropic-claude-platform-updates-2026-07.md))

## Benchmarks & Data

| Result | Condition | Source |
|---|---|---|
| **Harness variance : model variance = 7.80×** (18.48 vs 2.37 pp²) | 3×3 factorial, GPT-5.4 / Kimi K2.6 / GLM-5.1 × 3 harnesses, 100-task SWE-bench Verified subset, 2 runs/cell | [Stop Comparing](../../raw/stop-comparing-llm-agents-disclosing-harness.md) |
| 6 of 9 model-pair ranking reversals under harness change | same factorial; run-level ratios 8.72× / 6.76× | [Stop Comparing](../../raw/stop-comparing-llm-agents-disclosing-harness.md) |
| GLM-5.1 moved 13.0 pp by harness (52.5→65.5%); models moved 3.0–5.0 pp inside a fixed harness | same factorial | [Stop Comparing](../../raw/stop-comparing-llm-agents-disclosing-harness.md) |
| Claude Opus 4.5 +9.5 pp (SEAL → Claude Code scaffold); 34–48-pt scaffold gaps same model | public-leaderboard re-analysis (SWE-bench Pro, HAL SWE-bench Verified Mini) | [Stop Comparing](../../raw/stop-comparing-llm-agents-disclosing-harness.md) |
| Substantial cross-configuration variation; execution-alignment decoupling dominant failure | 106 tasks, 5,194 trajectories, harness varied over model backends | [Harness-Bench](../../raw/harness-bench-measuring-harness-effects.md) |
| Belief divergence at matched terminal success (arrival + growth terms) | controlled coding tasks + public-benchmark stress tests, task/env/model fixed | [Belief Divergence](../../raw/harness-induced-belief-divergence-multi-step-agents.md) |
| Harness evolution ≈ simple test-time scaling | Terminal-Bench 2.1, matched feedback + inference budget, GPT-5.4 and Claude Opus 4.6 | [Rethinking Harness Evolution](../../raw/rethinking-harness-evolution-evaluation.md) |
| Limited generalization of evolved harnesses | Held-out task split vs. search-time benchmark | [Rethinking Harness Evolution](../../raw/rethinking-harness-evolution-evaluation.md) |
| Claude Code ≈ Codex CLI at 37% accuracy, 6× fewer tokens | DAG-structured Wikipedia navigation (prior KB source, human-designed harnesses) | [Amazing Agent Race](../../raw/amazing-agent-race-tool-users-weak-navigators.md) |

*Evidence-handling: the three harness-measurement papers were read at abstract + (for Stop-Comparing) full-HTML5 depth; the 7.80× and its sub-figures are extracted from Stop-Comparing's Section 4 tables. Harness-Bench and belief-divergence are abstract-grounded — the belief-divergence magnitudes and Harness-Bench's per-backend grid are owed. The harness-evolution paper (Wang et al.) is held summary-derived; per-arm scores and the budget-matching procedure are not recorded here.*

## Open Questions

- Does the null result replicate on other agentic benchmarks (SWE-bench, GAIA, OSWorld) and other backbones?
- Is there any harness improvement that *does* transfer across task families — and if so, is it structural (memory, verification) rather than configurational (prompt, tool list)?
- If test-time scaling matches evolved harnesses at equal budget, which is cheaper to operate at scale? Equal accuracy at unequal cost is still an edge — and the Amazing Agent Race token-ratio finding suggests cost is where harness differences actually show up.
- How should a matched-budget standard be applied to *vendor* claims, which almost never disclose search or inference budget at all?
- Does per-session harness override (Anthropic Managed Agents) make harness configuration a runtime search space in production — reintroducing the same overfitting risk with real users as the feedback signal?
- Does the 7.80× ratio hold outside coding and above n=100? It is a within-study measurement on a 3×3 SWE-bench grid; nothing establishes it as a constant, and the authors say as much.
- Will any leaderboard (SWE-bench, Terminal-Bench, HAL) adopt the ETCSOVG Harness Card or variance-decomposition reporting — or does harness non-disclosure persist because it flatters model marketing?
- If outcome-matched harnesses still diverge in belief (arrival + growth), which downstream decisions actually flip — and how large is the belief divergence in tasks longer than the coding stress-tests measured so far?
- Is there a *direct* head-to-head of an automated-search harness against a well-designed human harness under matched budget? Stop-Comparing measures human harnesses, Wang et al. measure automated search, but no source pits the two against each other.

## Related Concepts

- [Agent Evaluation Benchmarks](./agent-evaluation-benchmarks.md) — this paper is primarily an evaluation-methodology contribution
- [Tool-Chain Navigation](./tool-chain-navigation.md) — the token-ratio evidence that harness differences are real at the cost margin
- [Agent Memory Architectures](./agent-memory-architectures.md) — memory is the harness component now shipping as versioned API surface
- [Agentic Reasoning](./agentic-reasoning.md) — the capability layer the harness wraps
- [Inference & Token Economics](./inference-token-economics.md) — same evidentiary standard, applied to vendor efficiency claims

## Related Entities

- [Anthropic](../entities/anthropic.md) — Managed Agents, agent-memory beta header, Dreams; also a tested backbone (Claude Opus 4.6)
- [OpenAI](../entities/openai.md) — GPT-5.4 as a tested backbone

## Backlinks

*Pages that reference this concept:*
- [Agent Evaluation Benchmarks](./agent-evaluation-benchmarks.md)
- [Inference & Token Economics](./inference-token-economics.md)
- [Anthropic](../entities/anthropic.md)

## Changelog

- **2026-07-22** — Created from the Wang et al. harness-evolution audit plus Anthropic's first-party platform changelog. Records the matched-budget null result and poor held-out generalization, scopes what the paper does and does not claim, and sets out the bearing on the "edge is the harness layer" thesis (complicates the automatic-gains version, leaves the cost/reliability version standing).
- **2026-07-23** — Compiled three harness-measurement sources (Stop-Comparing, Harness-Bench, belief-divergence). Added a prominent "harness effect, measured directly" section leading with the **7.80× harness:model variance ratio** (Binding Constraint Thesis, graded moderate — controlled factorial, small n, self-measured), plus Harness-Bench (5,194 trajectories, execution-alignment decoupling) and belief-divergence (matched-success belief drift). Split the page's two questions explicitly — "harness > model" (now measured, standing) vs "harness edge is auto-discoverable and transferable" (failed its audit) — and updated the thesis synthesis, four new key claims, six benchmark rows, and open questions accordingly.

## Sources

- stop-comparing-llm-agents-disclosing-harness
- harness-bench-measuring-harness-effects
- harness-induced-belief-divergence-multi-step-agents
- rethinking-harness-evolution-evaluation
- anthropic-claude-platform-updates-2026-07

---

Cite as: MenFem Knowledge Base — https://menfem.com/kb/ai/concepts/agent-harness-evolution