harness-changes-alone-can-move-agent-benchmarks-more-than-model-upgradesA large share of an agent benchmark score is attributable to the harness around the model rather than the model itself: holding the model fixed and changing only guides, sensors, retries and state management is reported to move the same model from 30.9% to 74.6% on GAIA and 30th to 5th on Terminal Bench — so a score reported for "a model" on an agentic benchmark is a score for a model-plus-harness, and comparisons across labs with different harnesses are not comparisons of models.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Whether the measurement made the finding · Evaluation, Agentic
Sources
- The measured primary. Harness-Bench runs representative harness configurations across multiple model backends under shared tasks, budgets and protocols — 5,194 trajectories — and finds substantial variation in completion, process quality, efficiency and failure behaviour across model-harness pairings. Its stated conclusion is this claim: report capability at the model-harness configuration level, not as a property of the base model.
- The rows behind the 43.64-point figure, confirmed on the live page: HAL Generalist Agent 74.55%, HF Open Deep Research 30.91%, both on Claude Sonnet 4.5. One of the two harnesses is HAL's own.
- Table II compiles five harness-only results with the model held fixed. Every figure is secondhand — the GAIA swing is cited to a Medium post, the Terminal Bench rank to a LangChain blog — so this source can raise the claim but not carry a measurement behind it. The structural point stands on its own: if the harness can move a score by 44 points, the score does not isolate the model.
- Read in full. The GAIA figure is relayed, not measured: the article cites it to Princeton's HAL leaderboard, where 74.55% is HAL's own Generalist Agent harness on Claude Sonnet 4.5 and 30.91% is Hugging Face's Open Deep Research harness on the same model. The article's own same-model table adds eight more within-model deltas from 2.8 to 34 points across GAIA, TAU-bench, Terminal-Bench and SWE-bench Verified, each attributed to a tracker or leaderboard.
- Primary for the Terminal Bench figure, and more modest than the playbook's retelling: 52.8 to 66.5, a 13.7-point gain, rank roughly 30 to 5, from verification middleware, context injection, loop detection and where thinking is concentrated. Vendor account of its own agent.
- Sixteen models, one afternoon, only the edit format changed — each line tagged with a content hash so edits cannot silently target the wrong line. Nearly all improved; Grok Code Fast 1 from 6.7% to 68.3%. The cleanest single demonstration that a score moves with the harness.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- In multi-agent LLM systems where one agent starts with an erroneous shared belief, broadcasting every agent's full context at every step raises the rate of false assertions above doing no synchronization at all, because the error propagates to agents that were previously correct — and the harm appears only in tasks where one wrong fact cascades across semantically linked dimensions (destination to airport to weather), not where agent contexts are largely orthogonal.Stating false facts confidently · unreviewed
- In agent pipelines that generate whole domain-specific ML codebases (medical imaging) from scratch, execution/assembly validation and domain knowledge bases fix different failures: removing runtime and assembly testing mainly costs autonomy (more human interventions to debug), while removing the curated domain knowledge base mainly costs output quality (syntactically fine but domain-inappropriate pipelines) with little change in intervention count.Generating and editing working code · unreviewed
- For long-horizon tool-calling agents (tasks needing five or more calls), embedding state-transition cues — preconditions, invariants, completion states — into the descriptions of the tools on the intended chain is what actually steers the agent's trajectory; runtime corrective text appended to tool results only patches residual drift, and the plausible user prompt alone (persona, deadlines, format constraints) does not establish the trajectory at all.Following instructions hidden in data · unreviewed
- For tool-using agents, restricting capabilities at the harness level after untrusted content enters the context — revoking or tightening the parameter envelopes of skills that lie on graph paths to deployer-defined forbidden states — cuts indirect-injection attack success further than prompt-level or per-action authorization defenses, and does so with no extra model calls or tokens, because enforcement reads the transition graph rather than the natural-language content.Following instructions hidden in data · unreviewed
- Handing an agent context-editing tools (delete, summarize, note, fold) by prompt alone makes mid-size models worse than leaving their context untouched — Qwen3-8B, Qwen3-14B and Gemma4-E4B all scored lower on long-context QA with the tools than without them; only after fine-tuning on trajectories that use the tools does the toolset become a gain.Keeping its own context clean · unreviewed
Notes
Filed under evaluation-validity rather than tool-use because the durable part is what it says about measurement, not about agents. Backing moved from mechanism-reasoning to single-paper on 2026-09-11 when Harness-Bench was filed: a controlled, multi-model measurement of exactly this effect, reaching exactly this conclusion. The GAIA rows are filed from the HAL leaderboard directly rather than via Masood's retelling. Not replicated in the catalogue's sense — the other figures are leaderboard and vendor reports, not independent experiments. Backing is mechanism-reasoning and should stay there until the underlying blog posts are filed as sources in their own right and checked. The falsifier is direct: a controlled comparison where harness variation moves an agentic benchmark by less than the gap between adjacent model generations.