Harnesses & Agent Systems

A harness is the machinery wrapped around a model: what it sees, which tools it can call, how its context is managed, what gets verified, what gets logged, and what happens when something fails. Lilian Weng's definition is the cleanest one on the rung — "the system surrounding a base model that orchestrates execution and decides how the model thinks and plans, calls tools and acts, perceives and manages context, stores artifacts, and evaluates results" (Harness Engineering for Self-Improvement).

What this rung prices

Everything above sits between a token price and a task price. A model quote is per token; a buyer pays per finished job, and the harness decides how many tokens a finished job takes and how often the job finishes at all. That gap is measurable: hold the task, the budget and the model fixed, change only the harness, and success rates move further than swapping the model does. So the rung answers a commercial question, not an academic one — if the harness is the binding constraint, then buying a better model and building a better harness are not substitutes, and the cost per completed task is a property of the pairing rather than of the model on the invoice.

The load-bearing findings

• Harness variance beat model variance by 7.80× in the one controlled factorial anyone has run. Three frontier models clustered near 45% on a coding leaderboard, crossed with three harness configurations over a 100-task SWE-bench Verified subset: average harness-induced variance 18.48 pp² against average model-induced variance 2.37 pp², with model-pair ranking reversals in 6 of 9 comparisons. Harness changes moved GLM-5.1 by 13.0 points; model swaps inside a fixed harness moved results 3.0–5.0 points (Stop Comparing LLM Agents Without Disclosing the Harness). This is the binding constraint thesis, and the same paper proposes the ETCSOVG Harness Card as the disclosure fix.
• Automatic harness evolution does not survive a matched budget. Harness search is itself test-time scaling — it spends inference to find a better scaffold — so it has to be compared against plain search at equal budget. Done that way on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6, evolved harnesses did not consistently beat simple test-time scaling, and generalised poorly to held-out tasks (Rethinking the Evaluation of Harness Evolution for Agents; this close read is abstract-level, so the claim rests on the abstract, not the paper body). Prime Intellect's Prime Agent claims wins across eleven benchmarks and is the live counter-case — but it publishes no matched-budget control, changes both baseline and backbone per row, and loses LongBench v2 to Claude Code (0.714 vs 0.746). Its 95.5% ARC-AGI-3 result sits 0.1 points above the human-expert baseline, which is a tie reported as a win.
• Agents are strong tool users and weak navigators — the failure is composition, not capability. Across 1,400 DAG-structured Wikipedia tasks the best agent reached 37.2% accuracy; navigation errors accounted for 27–52% of failures while tool-use errors stayed below 17%, and the pit-stop visit rate fell 13–18 points moving from linear to compositional structures while roadblock completion held steady (The Amazing Agent Race). The same run found Claude Code matching Codex CLI at 37% with 6× fewer tokens — token efficiency and task accuracy come apart, which is exactly the token-price-to-task-price wedge.
• The dominant failure class lives in the execution loop, not in the model's knowledge. Harness-Bench ran 5,194 trajectories over 106 sandboxed tasks with the task, budget and evaluation protocol held fixed, and named "execution-alignment failures, where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts" (Harness-Bench, read at abstract level). Its conclusion is attributional: report performance for model-harness configurations, never for models alone.
• Two harnesses can post the same score and leave the agent believing different things. Blocked actions, compressed repairs, selective verification and cost-aware evidence pruning often preserve terminal success while shifting the agent's beliefs about progress, risk, recoverability and next action (Harness-Induced Belief Divergence). Outcome parity is therefore not harness equivalence — see agent belief divergence. This ingest is abstract-only and carries no magnitudes.
• Memory is the harness's job because weights are an unreliable place to put facts. After 20 sequential weight writes, bare-statement facts retained 1% accuracy against 46% for facts drawn from broad study data; 70% of wrong answers contained the most recently written fact, and a forgotten fact re-supplied in the prompt recovered to 77–80% (Can a Language Model Learn Facts Continually in Its Weights?). Vendor-published and Qwen3-only, but it is the empirical backing for treating the file system as persistent memory rather than a convenience.
• Persistent state is the attack surface, and one framing trick opens it. On a deployed personal agent, poisoning any single dimension of Capability / Identity / Knowledge lifted attack success from a 24.6% baseline to 64–74%, and file protection that blocked 97% of malicious injections also blocked 93% of legitimate updates (Your Agent, Their Asset). Separately, across roughly 10,000 trials nine of twelve hypothesised attack dimensions produced no detectable exploitation — but goal reframing did, at 38–40% on Claude Sonnet 4, while GPT-4.1 exploited nothing across 1,850 trials (Mapping the Exploitation Surface).

What we do not know yet

• Nobody here has run the experiment. Every number on this rung is read it standing. The measurement the rung needs — the same task set at the same inference budget, a self-improving harness against a plain one — has not been published by anyone and has not been run here. verifiers v1 makes the separation of taskset / harness / runtime architectural, and Pi is a configurable minimal-vs-extended harness — both are instruments for that run, and neither publishes a benchmark of its own.
• Which harness features cause which gains. Harness-Bench measures that the pairing matters; it does not decompose the effect into context management, retry policy, verification or tool surface. The ETCSOVG Harness Card is a proposed disclosure standard with almost no worked examples and no leaderboard has adopted it.
• Whether harness benefit is really non-monotonic in model tier. The survey reports Lin et al. finding harness updating flat from Qwen2-32B to Opus 4.6 while harness benefit peaks at middle-tier models. That is the single most decision-relevant claim on the rung and it reaches us second-hand — the survey is a secondary source and none of its figures were verified against their primaries.
• How far the 7.80× ratio travels. It comes from a 3×3 grid, 100 coding tasks, two runs per cell. It is a within-study measurement, not a constant, and it has not been reproduced outside coding.
• Whether self-modifying harnesses can be kept honest. Prime Agent's own release reports reward hacking — it found a way to bypass Factorio's rules. A system that can edit its own scaffolding can edit its way around the objective, and no source here offers a control for that.

Read next

• stop-comparing-llm-agents-disclosing-harness — the thesis and the only controlled factorial.
• rethinking-harness-evolution-evaluation — the matched-budget check that deflates most harness-evolution claims.
• harness-engineering-for-self-improvement — the map of the whole field, caveats included.