Harnesses — Frontier

The open questions on the rung that turns a token price into a task price. Compiled from the 15 sources moved here when the rung was split out of models on 2026-08-05; updated 2026-09-24 when the rung reached 23 sources and first held cost-per-task measurements.

Active questions

Question	State of evidence	What would settle it
Does the harness advantage survive at matched budget?	Partly answered, and the answer is uncomfortable. Harness-induced variance is 7.80× model-induced variance — but automatic harness evolution does not beat simple test-time-scaling under matched feedback/inference budget. So harness design matters enormously while harness self-improvement may be mostly unmatched spend.	Matched-budget replications of self-improving scaffolds on held-out task families.
Is compositional reasoning a model limit or a harness limit?	Open, and it is the field's central bottleneck. Best agents reach 37.2% on 1,400 navigation tasks with navigation errors at 27–52% of failures — per-step tool use is fine, step selection is not. Explicit problem-model construction helps against CoT and ReAct, which points at the harness.	Whether frontier model releases move the navigation number at all, or only the per-step numbers.
How model-dependent is the exploitation surface?	Wide open, and consequential. Goal reframing was the sole reliable trigger of 12 dimensions — yet GPT-4.1 was completely immune across 1,850 trials. If immunity is reproducible, "agents are exploitable" is the wrong granularity of claim.	Cross-model replication of the 12-dimension battery on current frontier models.
Does memory curation pay for itself?	Under-measured. The write-manage-read frame makes manage the decisive stage, and A-MEM shows a design, but the KB holds no cost-per-task figure for curation against a naive append-and-retrieve store.	A matched-budget comparison of curated vs append-only memory on a long-horizon benchmark.
What is the actual cost-per-completed-task spread across harnesses?	No longer empty — corrected 2026-09-24. This row used to say nothing here reported tokens or dollars per finished job. Three sources now do. In tokens, with the model fixed: up to ~40× tokens per solved task across three open-source harnesses (Goose ~28–37K vs OpenCode ~1.15–1.55M) while paired pass rates move 0–8 points, and a model swap moves it only 1.0–1.3× (Scaffold Effect). In dollars, vendor-run: $0.21 → $0.12 per task (−41%) across six models (The Harness Effect). In context: trimmed history at 553,374 tokens beat full history at 1,480,996 on completion, 91.6% vs 71.0% (Less Context). See Cost per Completed Task.	The open-source 40× priced in dollars at a stated cache-hit rate, on a non-coding task set, including a closed commercial agent.

What this rung does not yet hold

• No cost-per-task measurement of our own. (Corrected 2026-09-24: this line used to say the rung held no cost-per-task measurement at all; it now holds three, all read — see the last row above.) None has been run here, and the gap most worth closing is still the one our own ran it seat can close where a literature review cannot: the same task across harnesses, tokens and dollars to completion. (Added 2026-08-18: DeepSeek Harness — MIT, TypeScript, developer preview, agent loop itself a swappable plugin — is the first widely-available artifact that could be instrumented against this directly: same task, same model, loop swapped. It is a release record here, not an ingest; nothing has been run against it.)
• No production-scale failure data beyond the OpenClaw deployment study — the evidence base is dominated by offline benchmarks.
• Nothing on human-in-the-loop harnesses — approval gates, escalation policy, supervision cost. Given how MenFem's own desks operate, that absence is conspicuous.

Standing note

This is the one rung where our seat is ran it — we operate desks, subagents and workflows daily. Per the teaching doctrine that is the standing a teaching-level claim requires, and the cost-per-task gap above is precisely where running it beats reading about it. The converse binds too: a claim here we have not run is coverage, not teaching.