harness-changes-alone-can-move-agent-benchmarks-more-than-model-upgrades CLM-0069mechanismsingle paperReviewed by AI
A large share of an agent benchmark score is attributable to the harness around the model rather than the model itself: holding the model fixed and changing only guides, sensors, retries and state management is reported to move the same model from 30.9% to 74.6% on GAIA and 30th to 5th on Terminal Bench — so a score reported for "a model" on an agentic benchmark is a score for a model-plus-harness, and comparisons across labs with different harnesses are not comparisons of models.
Evidence for: Simon's law (holds)
Capability: Whether the measurement made the finding · Evaluation, Agentic
Sources
- The measured primary. Harness-Bench runs representative harness configurations across multiple model backends under shared tasks, budgets and protocols — 5,194 trajectories — and finds substantial variation in completion, process quality, efficiency and failure behaviour across model-harness pairings. Its stated conclusion is this claim: report capability at the model-harness configuration level, not as a property of the base model.
- The rows behind the 43.64-point figure, confirmed on the live page: HAL Generalist Agent 74.55%, HF Open Deep Research 30.91%, both on Claude Sonnet 4.5. One of the two harnesses is HAL's own.
- Table II compiles five harness-only results with the model held fixed. Every figure is secondhand — the GAIA swing is cited to a Medium post, the Terminal Bench rank to a LangChain blog — so this source can raise the claim but not carry a measurement behind it. The structural point stands on its own: if the harness can move a score by 44 points, the score does not isolate the model.
- Read in full. The GAIA figure is relayed, not measured: the article cites it to Princeton's HAL leaderboard, where 74.55% is HAL's own Generalist Agent harness on Claude Sonnet 4.5 and 30.91% is Hugging Face's Open Deep Research harness on the same model. The article's own same-model table adds eight more within-model deltas from 2.8 to 34 points across GAIA, TAU-bench, Terminal-Bench and SWE-bench Verified, each attributed to a tracker or leaderboard.
- Primary for the Terminal Bench figure, and more modest than the playbook's retelling: 52.8 to 66.5, a 13.7-point gain, rank roughly 30 to 5, from verification middleware, context injection, loop detection and where thinking is concentrated. Vendor account of its own agent.
- Sixteen models, one afternoon, only the edit format changed — each line tagged with a content hash so edits cannot silently target the wrong line. Nearly all improved; Grok Code Fast 1 from 6.7% to 68.3%. The cleanest single demonstration that a score moves with the harness.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When an agent harness is evolved from execution traces under a fixed interaction budget, verifying each candidate edit on tasks chosen for the behavior it targets and accepting it only when the gain is attributable to that behavior beats scoring every candidate on a shared fixed, random or coreset task split, and larger verification budgets do not compensate: fixed-split methods using up to 24x the rollouts fell below the unmodified harness in half of their harness-benchmark pairs.Improving its own scaffolding from failures · reviewed by AI
- When an agent system evolves its own harness from failure traces, batching failures across tasks before editing buys efficiency but not better failure attribution: pooled across three agent benchmarks, the share of harness edits that merely accommodate the current model rose from 60.0% under per-failure serial editing to 75.6% under cross-task aggregation, and only fell to 45.5% when a separate analyst/critic/engineer refinement stage attributed each recurring pattern before the edit was written.Improving its own scaffolding from failures · reviewed by AI
- When an automated search over agent-harness mechanisms is gated on token efficiency with only a tolerance band on capability, the resulting full stack buys large token savings at a small but consistent score loss, and scores below the best single discovered mechanism — on EdgeBench the four-mechanism stack cut token traffic 44.7-49.0% and cost about a third while scoring 42.0 vs 44.8 for the base harness on GPT-5.6 Sol, whereas adding ObservationPack alone scored 47.2.Improving its own scaffolding from failures · reviewed by AI
- Evolving an agent harness on 2,000 executable tasks disjoint from the evaluation benchmarks, by contrasting successful and failed trajectories on the same task to find recurring deficiencies and evolving five harness modules (agent loop, tool use, observation management, context management, completion detection) separately before integrating them, raised DeepSeek-V4-Flash from 47.6 to 52.4 on TerminalBench 2.0 and 73.4 to 76.5 on SWE-Bench Verified, transferred across domains and to other backbones frozen (GLM-5.2 59.6 to 61.8, MiniMax-2.5 41.6 to 44.9), while evolving the whole harness at once or without modules fell below the unevolved baseline (46.4 and 44.2).Improving its own scaffolding from failures · reviewed by AI
- On four diagram-grounded geometry benchmarks a multi-agent pipeline consistently helps open-source vision-language models (Qwen-2.5-VL 7B +6.8 and 32B +3.3 points on Geometry3K, with further gains on OlympiadBench and We-Math) but a strong closed model (Gemini 2.0 Flash) generally did better as a single agent on the classic benchmarks, gaining only modestly on the newest one, so agentic decomposition pays for weaker models and less familiar tasks rather than universally.Getting more from several models than from one · reviewed by AI
Notes
Filed under evaluation-validity rather than tool-use because the durable part is what it says about measurement, not about agents. Backing moved from mechanism-reasoning to single-paper on 2026-09-11 when Harness-Bench was filed: a controlled, multi-model measurement of exactly this effect, reaching exactly this conclusion. The GAIA rows are filed from the HAL leaderboard directly rather than via Masood's retelling. Not replicated in the catalogue's sense — the other figures are leaderboard and vendor reports, not independent experiments. Backing is mechanism-reasoning and should stay there until the underlying blog posts are filed as sources in their own right and checked. The falsifier is direct: a controlled comparison where harness variation moves an agentic benchmark by less than the gap between adjacent model generations.