Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents

Abstract

Verbatim: "Software-agent benchmarks usually report whether an agent solves a task, but the agent reaches that outcome through a harness that controls what it sees, which actions it can take, which failures are repaired, which states are verified, and which evidence is logged. We show that this harness can change the agent's multi-step beliefs even when the task, environment, and base LLM are fixed. We introduce a belief-rollout diagnostic that elicits structured K-step trajectories over progress, risk, recoverability, constraints, failure mode, uncertainty, future success, repair cost, and next action under alternative harnesses. We define a cross-harness belief divergence and decompose it into an arrival term for immediate interface shifts and a growth term for horizon-dependent belief changes. On controlled coding tasks and public-benchmark stress tests, blocked actions, compressed repairs, selective verification, and cost-aware evidence pruning often preserve terminal success while changing the beliefs that drive later decisions. We further introduce BIWM, a no-training protocol that canonicalizes observations, logs censored branches, expands repair traces, records verification masks, executes risky branches in shadow, and aligns belief trajectories across harness views. The results suggest that harness design is an experimental variable in agent evaluation, not an implementation detail."

Key Contributions

• Belief-rollout diagnostic — elicits structured K-step belief trajectories over nine dimensions (progress, risk, recoverability, constraints, failure mode, uncertainty, future success, repair cost, next action) under alternative harnesses on a fixed task/env/model.
• Cross-harness belief divergence, decomposed into an arrival term (immediate belief shift from the interface change) and a growth term (horizon-dependent belief drift accumulated over steps).
• BIWM, a no-training alignment protocol: canonicalizes observations, logs censored branches, expands repair traces, records verification masks, executes risky branches in shadow, and aligns belief trajectories across harness views.
• Finding: common harness moves — blocked actions, compressed repairs, selective verification, cost-aware evidence pruning — often preserve terminal success while altering the beliefs that drive downstream decisions. Terminal-outcome parity hides mid-trajectory belief divergence.

Methodology

Fix task, environment, and base LLM; vary the harness; run the belief-rollout diagnostic to obtain K-step structured belief trajectories per harness; compute cross-harness divergence and split it into arrival vs growth terms. Evaluated on controlled coding tasks plus public-benchmark stress tests. BIWM is applied as a no-training normalization layer so trajectories are comparable across harness views.

Results

No numeric benchmark table is present in the abstract. The qualitative result is that terminal success is frequently preserved across harness variants while intermediate beliefs diverge — establishing harness design as an experimental variable, and belief divergence (not just pass/fail) as the thing to measure. (Full-text quantitative results not captured in this ingest.)

Limitations

Abstract-only numeric grounding at ingest — the magnitude of belief divergence and the arrival/growth split are not quantified here. Small author team, diagnostic scope, coding-task-centric. The contribution is an instrument and a framing, not a fix.

Relation to the KB

The freshest (Jul 5, 2026) of this discovery pass's harness cluster and a distinct angle: where Stop Comparing measures harness effects on terminal performance and Harness-Bench measures them on outcomes/failure behavior, this measures them on the agent's internal beliefs. Sharpens frontier §5 (Harness Layer) and §12 (Evaluation Standardization): even outcome-matched harnesses are not interchangeable.

Source: arXiv:2607.04528 — Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents, Yi & Song, 5 July 2026. Abstract page retrieved 2026-07-23.