The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI

Conflict of interest, stated up front. The harness measured is the authors' own (the Writer Agent Harness), and one of the six models tested (Palmyra X6) is also theirs. This is a vendor-run controlled swap. What partly offsets it: quality is held as a reported metric rather than assumed away, and the efficiency claim is made across five models the vendor does not own. Cite the shape of the result confidently; cite the magnitude as vendor-reported.

Abstract (verbatim)

"Agentic AI development today runs on token maxing: buying capability with tokens -- longer reasoning traces, more turns, wider tool payloads, bigger replayed contexts -- so tokens per task grow faster than task value. Falling per-token prices mask the pattern; total spend rises anyway. We argue the decisive lever against token maxing is the harness: the orchestration layer that assembles context, exposes tools, sequences turns, delegates work, and carries enterprise observability and governance. We isolate it with a controlled swap: 22 locked evaluation tasks, six foundation models (Claude Sonnet 4.6, Gemini 3.1, Gemini Flash 3.5, Qwen 3.6, GLM 5.1, Palmyra X6), changing only the orchestration layer -- a frozen conventional production loop versus the Writer Agent Harness. Holding models constant, the harness cuts blended cost per task 41% ($0.21->$0.12), median wall-clock 44% (48s->27s), and tokens per task 38% (14.2k->8.8k), with task-completion quality at parity (0.78->0.81, directional at this sample size). Efficiency is model-invariant -- every model gets cheaper (33-61%) -- while quality gains are capability-dependent: a model's gain correlates almost perfectly with its baseline strength (r=0.99, n=6), a phenomenon we term harness leverage. Quality per dollar rises 82%; task-completions per million tokens rise from 54.9 to 92.0. On this workload the orchestration layer moved cost per task more than the full spread of the model menu did. We formalize token economics at the orchestration layer (including effective input price under prompt caching), detail the six mechanism families behind the effect -- cache-shape discipline to failure-spend governance -- compare six widely used agent systems on the same axes, and argue the harness is the one component whose efficiency multiplies across every model an organization runs -- present and future."

Key contributions

1. "Token maxing" named as the failure mode. Capability is bought with tokens — longer traces, more turns, wider tool payloads, replayed context — so tokens per task grow faster than task value. Falling per-token prices mask this: unit price down, total spend up. This is the rung's through-line stated as a pathology.
2. The harness isolated by controlled swap. Models and tasks frozen; only the orchestration layer changes. That is the design that lets the paper attribute the delta to orchestration rather than to model choice.
3. Harness leverage. Efficiency gains are model-invariant (every model gets cheaper); quality gains are capability-dependent, correlating with baseline model strength at r=0.99 (n=6). A better harness does not rescue a weak model — it compounds a strong one.
4. Effective input price under prompt caching formalised — the price that matters once cache hits are priced differently from fresh input tokens, which is the accounting most cost-per-task estimates get wrong.
5. Six mechanism families behind the effect, spanning cache-shape discipline through failure-spend governance (i.e. the cost of runs that fail is itself a governed budget line).

Methodology

• 22 locked evaluation tasks (locked = fixed before the swap, not re-tuned per arm).
• Six foundation models: Claude Sonnet 4.6, Gemini 3.1, Gemini Flash 3.5, Qwen 3.6, GLM 5.1, Palmyra X6.
• Two arms: a frozen conventional production agent loop vs the Writer Agent Harness.
• Metrics: cost per task, tokens per task, median wall-clock, task-completion quality, quality per dollar, task-completions per million tokens.

Results

The number that matters for this rung: 54.9 → 92.0 completions per million tokens. That is a token→task conversion rate, not a token price — the exact unit the KB's through-line asks every rung to move.

Limitations

• Vendor-run. The winning harness and one model are the authors' own. Not a neutral bench.
• Quality is explicitly "directional at this sample size" — 22 tasks is too few for the 0.78→0.81 delta to carry weight. The paper says so; the efficiency claim, not the quality claim, is what it stands on.
• One workload. "On this workload" is the authors' own hedge. Generality across task classes (research, coding, long-horizon ops) is not established here.
• No cost breakdown by mechanism family is given in the abstract — which of the six mechanisms produced most of the 41% is unresolved without the full text.

How it bears on MenFem

• ai T1 — "agentic reasoning will consolidate around a standard stack." If orchestration dominates the cost curve and its gains are model-invariant, the harness is the layer that standardises, and the model menu becomes the swappable part. This is direct evidence for that thesis, from the cost side.
• Pairs with H1 (the 40x token-spread source, unpicked as of 2026-09-11). H1 states the harness dominance in tokens; this one states it in dollars per completed task with quality held. Same claim from both ends of the token→task bridge.
• The teaching moment is the masking effect: per-token prices fell and spend rose anyway. That is the counter-intuitive shape the doctrine wants, and it is the sentence a reader keeps.