Harnesses & Agent Systems

Three-layer framework for agentic reasoning: foundational, self-evolving, multi-agent

Unified taxonomy of ~60 benchmarks, agent framework comparison, collaboration protocols survey

Unified evolutionary framework for LLM tool use: prompting, supervised, RL paradigms

Red-teaming framework for agentic AI: permission escalation, hallucination, orchestration flaws, memory manipulation, supply chain

Write-manage-read taxonomy, 5 mechanism families, three-dimensional taxonomy for agent memory

Agentic memory with Zettelkasten-inspired note construction, dynamic linking, memory evolution

Large-scale systematic taxonomy of LLM agent exploitation triggers across 12 attack dimensions, identifying goal reframing as the sole reliable trigger while ruling out nine others, with GPT-4.1 achieving complete immunity across 1,850 trials.

First real-world safety evaluation of a deployed personal AI agent (OpenClaw), introducing the CIK taxonomy and showing that poisoning any single dimension raises attack success rate from 24.6% to 64–74%.

DAG-structured benchmark of 1,400 Wikipedia navigation tasks revealing that current best agents achieve only 37.2% accuracy with navigation errors dominating (27–52% of failures), exposing compositional reasoning as the primary frontier bottleneck.

Two-phase reasoning: LLMs construct explicit problem models before generating solutions. Reduces constraint violations vs CoT and ReAct across five planning domains.

Automatic harness evolution for LLM agents does not consistently outperform simple test-time-scaling baselines under matched feedback/inference budget, and generalizes poorly to held-out tasks (Terminal-Bench 2.1, GPT-5.4 + Claude Opus 4.6) — a methodological check on 'scaffold gains' claims in the agentic-harness literature.

Formalizes the Binding Constraint Thesis — for long-horizon agentic tasks across comparable-capability models the execution harness is often a stronger performance determinant than the model — and measures harness-induced variance at 7.80x model-induced variance (18.48 vs 2.37 pp^2, 6/9 ranking reversals) in a 3-model x 3-harness SWE-bench Verified experiment; proposes an ETCSOVG Harness Card disclosure standard + variance-decomposition protocol. Directly on the MenFem 'edge is the harness' thesis.

Diagnostic benchmark (106 sandboxed offline tasks, 5,194 execution trajectories) that isolates configuration-level harness effects from model capability by fixing task/budget/eval and varying only the harness across model backends; finds substantial variation in completion, quality, efficiency, and failure behavior, and names execution-alignment decoupling as the dominant failure class. Empirical complement to Stop-Comparing.

Shows the harness changes an agent's multi-step beliefs (progress, risk, recoverability, constraints, failure mode, uncertainty, future success, repair cost, next action) even with task/env/model fixed; introduces a belief-rollout diagnostic, a cross-harness belief-divergence metric split into arrival (interface) + growth (horizon) terms, and BIWM (no-training trajectory alignment). Terminal success often preserved while decision-driving beliefs diverge. Freshest of the July harness cluster.

Controlled orchestration swap, 6 models x 22 locked tasks: cost/task $0.21->$0.12 (-41%), tokens/task 14.2k->8.8k (-38%), completions per Mtok 54.9->92.0, quality at parity (0.78->0.81, directional). Efficiency model-invariant (33-61%); quality gain correlates with baseline model strength r=0.99. VENDOR-RUN (Writer harness + Palmyra X6 are the authors' own).

50-task MCP tool-use benchmark, 5 runs: full history 71.0% completion at 1,480,996 tokens / 14.56h; last-5-tool-calls 79.0% at 535,274 / 5.39h; pruning+summarisation 91.6% (99.64% avg amount) at 553,374 / 5.79h. More context is both 2.68x more expensive AND 20.6 points worse. Cross-model check with Claude Sonnet 4.5.