In scope: scaffolding and orchestration, tool use, context and memory management, retries and recovery, agent evaluation and benchmarks, failure modes at step three, agent security and the exploitation surface, and cost per completed task. The load-bearing finding this rung already holds: for long-horizon agentic work the harness is often a stronger performance determinant than the model.
| Type | Source | Published |
|---|---|---|
| ANALYSIS | Pi — a minimal, extensible terminal coding harness Earendil Inc. and contributors · Earendil Inc. A configurable, inspectable harness — an INSTRUMENT for testing the binding-constraint thesis, not evidence for it. Publishes no measurement. | — |
| REPORT | AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms Google DeepMind · Google DeepMind Gemini-powered evolutionary coding agent; 0.7% Google compute recovery; math breakthroughs | 2025-05-14 |
| REPORT | Prime Agent: A self-improving RLM harness Prime Intellect · Prime Intellect Self-improving RLM harness with CRUD-editable harness state (prompts/skills/memory/sub-agents) in a persistent IPython kernel; reports 95.5% ARC-AGI-3 vs a 95.4% human-expert baseline (a 0.1pp margin) and wins on 9 of 11 rows. VENDOR SELF-REPORT with NO matched-budget control -- the exact control that would test agent-harness-evolution's negative result. Loses LongBench v2 to Claude Code 0.746 vs 0.714. Reward hacking observed (bypassed Factorio's rules). | 2026-08-05 |
| REPORT | verifiers v1: Decomposing Tasksets and Harnesses for Agentic RL & Evaluations Prime Intellect · Prime Intellect Decomposes the agent environment into composable taskset / harness / runtime, with an interception server normalising OpenAI Chat Completions, OpenAI Responses and Anthropic Messages while recording traces. Message-DAG traces cut storage from O(n^2) to O(n) in turns, making 100+ turn rollouts practical. Makes the matched-budget harness comparison architectural rather than bespoke. NOTE: actually v0.2.0, a PREVIEW under the verifiers.v1 namespace, released 2026-07-10 — the universal 'v1' framing in secondary coverage overstates maturity. No benchmark in the primary; the 'matched Harbor' and GLM-4.5-Air figures circulating second-hand do not appear in the release note. | 2026-07-10 |
| REPORT | Can a Language Model Learn Facts Continually in Its Weights? Charles O'Neill · Baseten Facts written to weights are question-keyed and do not survive later writes. After 20 sequential writes bare-statement facts retain 1% accuracy vs 46% for broad-study data; 70% of wrong answers contain the MOST RECENTLY written fact; a forgotten fact re-supplied in the prompt recovers to 77-80%. Context, not weights, is the reliable channel — the empirical case for memory being the harness layer job. | 2026-07-14 |