Rung 01 Harnesses & Agent SystemsSwitch rungClose
Cost per Completed Task
Active FrontierA model is priced per token. A buyer pays for a finished job. The number that joins the two is how many tokens it takes to finish one job — counting the runs that fail — and on this rung that number is set more by the harness than by the model.
Until 2026-09-24 this rung's own frontier said it held no cost-per-task measurement at all. That is no longer true: three sources now measure it, from three directions.
In tokens, with the model held fixed (the cleanest). Two models, 50 coding tasks, three open-source harnesses. Tokens per solved task ran from about 28,000–37,000 under Goose to about 1.15–1.55 million under OpenCode — roughly 40× — while pass rates stayed within 38–50% and paired differences within 0–8 points. Swapping the model moved tokens per solved task only 1.0–1.3×. The extra tokens were not extra turns; they were more tokens per turn, including "no-action" turns in which the agent neither edits nor runs anything but still re-reads its whole history (The Scaffold Effect in Coding Agents).
In dollars, vendor-run. Six models, 22 fixed tasks, only the orchestration layer swapped: cost per task fell 41% ($0.21 → $0.12), tokens per task 38% (14.2k → 8.8k), and completed tasks per million tokens rose from 54.9 to 92.0, with quality at parity. The winning harness and one of the models are the authors' own (The Harness Effect).
In context, one enterprise workflow. On 50 expense tasks, keeping the whole conversation cost 1,480,996 tokens for 71.0% completion; keeping only the last five tool exchanges plus a short summary cost 553,374 tokens for 91.6% — cheaper and more reliable, because old tool output misleads the agent (Less Context, Better Agents).
Put together: a buyer comparing models by price per token is choosing inside a narrow band; the harness choice sits on a band an order of magnitude wider. The Writer paper names the failure mode "token maxing" — buying capability with more tokens, masked by falling per-token prices.
Key Claims
- Harness choice moves tokens per solved task ~40×; a model swap moves it 1.0–1.3×. Pass rate moves 0–8 points. Evidence: moderate (workshop preprint, 300 trials, two models, open-source harnesses only) (Scaffold Effect)
- A controlled orchestration swap cut cost per task 41% across six models. Evidence: moderate (vendor-run, 22 tasks) (The Harness Effect)
- Trimming context raised completion from 71.0% to 91.6% while cutting tokens 63%. Evidence: moderate (one Microsoft workflow, 5 runs) (Less Context)
- Each harness fails in its own way whatever the model. Goose stops when stuck, OpenHands-SDK hands in plausible wrong answers or runs out of turns, OpenCode rarely hands in a wrong answer but more often runs out of time. Evidence: moderate (Scaffold Effect)
Benchmarks & Data
| Measurement | Low | High | Source |
|---|---|---|---|
| Tokens per solved task by harness (Qwen 3.6 Plus) | 28,142 (Goose) | 1,147,740 (OpenCode) | Scaffold Effect |
| Tokens per solved task by harness (MiniMax M2.5) | 36,950 (Goose) | 1,546,977 (OpenCode) | same |
| Cost per task, conventional loop → Writer harness | $0.12 | $0.21 | The Harness Effect |
| Tokens per benchmark, trimmed+summary → full history | 553,374 | 1,480,996 | Less Context |
The evals rung reports the same direction from the evaluator's side: a 33× dollar spread on
identical tasks from agent set-up alone
(AI evals are becoming the new compute bottleneck).
Open Questions
- The 40× is tokens, not dollars; the $0.21 → $0.12 is dollars but vendor-run. Nobody here has priced the open-source harness gap in dollars at a stated cache-hit rate.
- Which harness features produce the gap — context loading, tool-output packaging, retries? The Scaffold Effect authors suspect per-turn context growth but do not isolate it.
- Does the gap hold for closed commercial agents? None were tested.
Related Concepts
- The Binding Constraint Thesis — the harness beats the model on score variance; this page is the same claim on cost.
- Measuring Harness Effects — how to hold everything but the harness fixed.
Changelog
- 2026-09-24 — Created. Compiles the Scaffold Effect paper (ingested today) with the Harness Effect and Less Context sources ingested 2026-09-11. Corrects the frontier's claim that the rung held no cost-per-task measurement.