Harnesses & Agent Systems

A configurable, inspectable harness — an INSTRUMENT for testing the binding-constraint thesis, not evidence for it. Publishes no measurement.

Three-layer framework for agentic reasoning: foundational, self-evolving, multi-agent

Unified taxonomy of ~60 benchmarks, agent framework comparison, collaboration protocols survey

Unified evolutionary framework for LLM tool use: prompting, supervised, RL paradigms

Red-teaming framework for agentic AI: permission escalation, hallucination, orchestration flaws, memory manipulation, supply chain

Write-manage-read taxonomy, 5 mechanism families, three-dimensional taxonomy for agent memory

Agentic memory with Zettelkasten-inspired note construction, dynamic linking, memory evolution

Large-scale systematic taxonomy of LLM agent exploitation triggers across 12 attack dimensions, identifying goal reframing as the sole reliable trigger while ruling out nine others, with GPT-4.1 achieving complete immunity across 1,850 trials.

First real-world safety evaluation of a deployed personal AI agent (OpenClaw), introducing the CIK taxonomy and showing that poisoning any single dimension raises attack success rate from 24.6% to 64–74%.

DAG-structured benchmark of 1,400 Wikipedia navigation tasks revealing that current best agents achieve only 37.2% accuracy with navigation errors dominating (27–52% of failures), exposing compositional reasoning as the primary frontier bottleneck.

Two-phase reasoning: LLMs construct explicit problem models before generating solutions. Reduces constraint violations vs CoT and ReAct across five planning domains.

Automatic harness evolution for LLM agents does not consistently outperform simple test-time-scaling baselines under matched feedback/inference budget, and generalizes poorly to held-out tasks (Terminal-Bench 2.1, GPT-5.4 + Claude Opus 4.6) — a methodological check on 'scaffold gains' claims in the agentic-harness literature.

Formalizes the Binding Constraint Thesis — for long-horizon agentic tasks across comparable-capability models the execution harness is often a stronger performance determinant than the model — and measures harness-induced variance at 7.80x model-induced variance (18.48 vs 2.37 pp^2, 6/9 ranking reversals) in a 3-model x 3-harness SWE-bench Verified experiment; proposes an ETCSOVG Harness Card disclosure standard + variance-decomposition protocol. Directly on the MenFem 'edge is the harness' thesis.

Diagnostic benchmark (106 sandboxed offline tasks, 5,194 execution trajectories) that isolates configuration-level harness effects from model capability by fixing task/budget/eval and varying only the harness across model backends; finds substantial variation in completion, quality, efficiency, and failure behavior, and names execution-alignment decoupling as the dominant failure class. Empirical complement to Stop-Comparing.

Shows the harness changes an agent's multi-step beliefs (progress, risk, recoverability, constraints, failure mode, uncertainty, future success, repair cost, next action) even with task/env/model fixed; introduces a belief-rollout diagnostic, a cross-harness belief-divergence metric split into arrival (interface) + growth (horizon) terms, and BIWM (no-training trajectory alignment). Terminal success often preserved while decision-driving beliefs diverge. Freshest of the July harness cluster.

Controlled orchestration swap, 6 models x 22 locked tasks: cost/task $0.21->$0.12 (-41%), tokens/task 14.2k->8.8k (-38%), completions per Mtok 54.9->92.0, quality at parity (0.78->0.81, directional). Efficiency model-invariant (33-61%); quality gain correlates with baseline model strength r=0.99. VENDOR-RUN (Writer harness + Palmyra X6 are the authors' own).

50-task MCP tool-use benchmark, 5 runs: full history 71.0% completion at 1,480,996 tokens / 14.56h; last-5-tool-calls 79.0% at 535,274 / 5.39h; pruning+summarisation 91.6% (99.64% avg amount) at 553,374 / 5.79h. More context is both 2.68x more expensive AND 20.6 points worse. Cross-model check with Claude Sonnet 4.5.

Gemini-powered evolutionary coding agent; 0.7% Google compute recovery; math breakthroughs

Self-improving RLM harness with CRUD-editable harness state (prompts/skills/memory/sub-agents) in a persistent IPython kernel; reports 95.5% ARC-AGI-3 vs a 95.4% human-expert baseline (a 0.1pp margin) and wins on 9 of 11 rows. VENDOR SELF-REPORT with NO matched-budget control -- the exact control that would test agent-harness-evolution's negative result. Loses LongBench v2 to Claude Code 0.746 vs 0.714. Reward hacking observed (bypassed Factorio's rules).

Decomposes the agent environment into composable taskset / harness / runtime, with an interception server normalising OpenAI Chat Completions, OpenAI Responses and Anthropic Messages while recording traces. Message-DAG traces cut storage from O(n^2) to O(n) in turns, making 100+ turn rollouts practical. Makes the matched-budget harness comparison architectural rather than bespoke. NOTE: actually v0.2.0, a PREVIEW under the verifiers.v1 namespace, released 2026-07-10 — the universal 'v1' framing in secondary coverage overstates maturity. No benchmark in the primary; the 'matched Harbor' and GLM-4.5-Air figures circulating second-hand do not appear in the release note.

Facts written to weights are question-keyed and do not survive later writes. After 20 sequential writes bare-statement facts retain 1% accuracy vs 46% for broad-study data; 70% of wrong answers contain the MOST RECENTLY written fact; a forgotten fact re-supplied in the prompt recovers to 77-80%. Context, not weights, is the reliable channel — the empirical case for memory being the harness layer job.

Survey of ~39 refs. Lin et al. (2026): harness-UPDATING capability FLAT across Qwen2-32B->Opus 4.6 while harness BENEFIT is NON-MONOTONIC, middle-tier models gaining most. DGM: SWE-bench Verified 20%->50%, Polyglot 14.2%->30.7%. RE-Bench: agents 4x humans at a 2h budget but humans win at 8h and 32h.