The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation

How deeply this was read: the full paper, from the arXiv HTML version (v1), fetched 2026-09-24, including all six tables. Every number below is read from that text.

The date check (asked for at approval). arXiv's own abstract page says "Submitted on 8 Jun 2026", and the full text is stamped "arXiv:2607.22585v1 [cs.AI] 08 Jun 2026" (v1 timestamp: Mon, 8 Jun 2026 20:27:06 UTC). Yet the ID prefix 2607 and the listing's browse context (2026-07) put it in July. The mismatch is in arXiv's own record, not in the proposal. The submission date arXiv states, 2026-06-08, is used as date_published; the paper entered arXiv's listings under July 2026. There is only one version.

Status of the work: the authors call it "Preliminary work; under review at the 5th DL4C Workshop @ ICML 2026". Seven pages, a workshop paper, not a journal article. Both authors are at Sentient Labs; the paper does not say whether Sentient builds or sells a harness.

Abstract

Verbatim from arXiv:

Public leaderboards for coding agents typically rank systems by model name and pass rate, while the surrounding harness (the scaffold that issues tools, manages context, and decides when to stop) is often under-specified. Model-to-model comparison is valid when the harness is fixed; when it varies, performance and efficiency conflate model and scaffold effects. We evaluate Qwen 3.6 Plus and MiniMax M2.5 across three open-source harnesses (Goose, OpenCode, OpenHands-SDK) on a stratified 50-task subset of Terminal-Bench Pro. Harness choice induces up to a 40x difference in tokens per solved task, while paired within-model pass-rate differences remain 0-8 percentage points (95% paired-task bootstrap CIs include zero except for the largest gap). Failure fingerprints replicate across models (REASON for Goose, VERIFY/MAX_TURNS for OpenHands-SDK, idle-loop/TIME for OpenCode), indicating harness-level biases that are largely model-independent. For human-centered coding-agent evaluation, model name alone is an incomplete comparison unit: harness-model pairs determine real-world cost, latency, and oversight burden; no-action turns are a per-task wait tax, not just a token tax. We therefore recommend selecting harness-model pairs by pass rate under token/latency budgets, and reporting token usage, latency, and full harness specifications alongside any model comparison. We release anonymized configs, raw trial logs, aggregated snapshots, and analysis scripts.

Key Contributions

• The harness moves cost 40×; it barely moves the score. 300 trials (3 harnesses × 2 models × 50 tasks). Tokens per solved task run from ~28K to ~1.5M depending on the harness, while pass rates stay within 38–50%.
• Upgrading the model barely moves cost. Swapping one model for the other changes tokens per solved task only 1.0–1.3× (Table 6).
• Each harness fails in its own way, whichever model runs in it. Goose gives up cleanly when stuck; OpenHands-SDK tends to hand in a plausible wrong answer or run out of turns; OpenCode rarely hands in a wrong answer but more often runs out of time.
• "No-action turns" as a cost a person pays. A turn where the agent neither edits a file nor runs a new command still costs a full round-trip and re-reads the whole conversation so far. OpenCode averages about 2 such turns per task; Goose about 0.2–0.3.
• A reporting fix: compare harness–model pairs, and publish tokens per solved task, idle turns and the failure breakdown alongside pass rate.

Methodology

• Tasks: 50 of the 200 public Terminal-Bench Pro tasks, drawn before any run, spread across 8 kinds of work (bug fix, build, data, fresh implementation, ML, puzzle, security, system). Each is graded by its own automated test suite.
• Models: Qwen 3.6 Plus and MiniMax M2.5, both through the OpenRouter API at default sampling.
• Harnesses: Goose, OpenCode, OpenHands-SDK, all run through the Harbor evaluation framework.
• Held fixed: the task wording, the tests, one sandbox per task, a 900-second time cap, and a near-identical system prompt. Not held fixed (the thing studied): each harness's own tools, how much context it loads up front, and its internal retry and sub-agent logic. Goose and OpenHands-SDK had a 40-turn cap; OpenCode exposes no turn cap through Harbor, so only the time cap bounded it.
• Tokens per solved task: all tokens billed in a set-up (including failed runs) divided by the number of tasks it solved — "the cost a deployer absorbs per successful task".

Results

Pass rate (Table 2), 95% confidence ranges in brackets:

Harness	Qwen 3.6 Plus	MiniMax M2.5
Goose	48.0% [34.0, 62.0]	38.0% [24.0, 52.0]
OpenCode	50.0% [36.0, 64.0]	46.0% [32.0, 60.0]
OpenHands-SDK	50.0% [36.0, 64.0]	46.0% [32.0, 60.0]

Tokens per solved task (Table 3):

Harness	Model	Tokens per solved task	× Goose	Avg. turns
Goose	Qwen	28,142	1.0	17.96
Goose	MiniMax	36,950	1.0	25.25
OpenHands	Qwen	841,201	29.9	25.95
OpenHands	MiniMax	843,286	22.8	24.19
OpenCode	Qwen	1,147,740	40.8	21.71
OpenCode	MiniMax	1,546,977	41.9	27.46

• The gap survives the uncertainty check: Goose's upper bound (40–61K tokens) sits far below OpenCode's lower bound (733K–1.01M).
• It is not because OpenCode ran more turns (21–27 against Goose's 18–25). The authors put it down to more tokens per turn: growing context, how tool output is packaged, and each harness's own token accounting.
• Harness vs model (Table 6): pass rate 0–8 points (harness) vs 4–10 points (model); tokens per solved task 40× vs 1.0–1.3×; no-action turns 10× vs under 1.1×.
• By kind of task (both models pooled): on fresh implementation OpenHands-SDK solved 69% against 38% for the other two; all three solved 0% of system-configuration tasks.

Limitations

The authors list these themselves:

• Small. 50 tasks, so one task is 2 points; most pass-rate gaps are within noise.
• Token counts are not measured the same way for every harness. Goose reports only a total, not input and output separately. The 40× rests on totals, which Goose does report.
• Turn caps differ. OpenCode had no turn cap; the authors checked that a 40-turn cap would have cut only ~12–29% of its runs, mostly ones that were failing anyway.
• Three harnesses, two models, open-source only. No closed commercial agent was tested.
• Provider defaults can change between runs.

Added here: the paper reports tokens, not dollars. Turning 40× the tokens into 40× the bill assumes the same price per token, which the authors state as "all else equal".

How it bears on the through-line

This is the cleanest single number on the KB for the gap between a token price and a task price. Hold the model's token price fixed, change only the harness, and the tokens needed to finish one task move 40× while the chance of finishing it barely moves. A buyer choosing between models on price per token is choosing inside a 1.0–1.3× band; the harness choice sits on a 40× band. It pairs with The Harness Effect, which prices the same swap in dollars per task (−41%, vendor-run), and with the evals rung's cost-of-measuring source, whose 33× dollar spread from scaffold choice points the same way (AI evals are becoming the new compute bottleneck).

---

Source: The Scaffold Effect in Coding Agents by Naman Vats and Oleg Golev, Sentient Labs, arXiv:2607.22585v1 (submitted 2026-06-08).