agent-harness-evolved-execution-traces-under-fixed-interaction-budget CLM-0545mechanismsingle paperReviewed by AI
When an agent harness is evolved from execution traces under a fixed interaction budget, verifying each candidate edit on tasks chosen for the behavior it targets and accepting it only when the gain is attributable to that behavior beats scoring every candidate on a shared fixed, random or coreset task split, and larger verification budgets do not compensate: fixed-split methods using up to 24x the rollouts fell below the unmodified harness in half of their harness-benchmark pairs.
Capability: Improving its own scaffolding from failures
Observed on
Three coding-agent harnesses (OpenCode, Codex CLI, Pi) on four benchmarks, one model family (deepseek-v4-flash-preview), 30 TRAIN tasks, budget capped at 200 units; gains largest w.
Sources
- supportsVerify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware VerificationValidated by: Held-out pass@1 across 3 harnesses x 4 benchmarks against the unmodified harness and three trajectory-based evolution baselines (Self-Harness, Meta-Harness, HarnessFix); best or tied-best in 8 of 12 pairs, average +7.6-13.6%, never below baseline harness. Ablations on OpenCode replace the behavior-aware batch with fixed, random and RHO-based batches (all fall back to the initial harness score) and replace the attribution gate with an aggregate-metric gate (gains only on Retail).. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When an agent system evolves its own harness from failure traces, batching failures across tasks before editing buys efficiency but not better failure attribution: pooled across three agent benchmarks, the share of harness edits that merely accommodate the current model rose from 60.0% under per-failure serial editing to 75.6% under cross-task aggregation, and only fell to 45.5% when a separate analyst/critic/engineer refinement stage attributed each recurring pattern before the edit was written.Improving its own scaffolding from failures · reviewed by AI
- When an automated search over agent-harness mechanisms is gated on token efficiency with only a tolerance band on capability, the resulting full stack buys large token savings at a small but consistent score loss, and scores below the best single discovered mechanism — on EdgeBench the four-mechanism stack cut token traffic 44.7-49.0% and cost about a third while scoring 42.0 vs 44.8 for the base harness on GPT-5.6 Sol, whereas adding ObservationPack alone scored 47.2.Improving its own scaffolding from failures · reviewed by AI
- A large share of an agent benchmark score is attributable to the harness around the model rather than the model itself: holding the model fixed and changing only guides, sensors, retries and state management is reported to move the same model from 30.9% to 74.6% on GAIA and 30th to 5th on Terminal Bench — so a score reported for "a model" on an agentic benchmark is a score for a model-plus-harness, and comparisons across labs with different harnesses are not comparisons of models.Whether the measurement made the finding · reviewed by AI
- In hours-long ML development runs on Kaggle-style competitions, agent scaffolds each lock into one strategy policy — the single-loop CLI agent pivots far less than humans (9% vs 25% of transitions) and the evolutionary search agent far more (58%) with pivot gains cancelling out — and neither reopens an approach it had abandoned, even though it recovers scores after setbacks as well as top humans.Planning a sequence of actions before acting · reviewed by AI
- When RL-training a central-agent multi-agent harness with 1.5B-3B Qwen2.5 backbones on multi-hop QA, optimizing only the coordinating central policy and leaving sub-agents frozen scores higher than jointly training central and sub-agent policies (average F1 47.79 vs 45.36, accuracy 42.88 vs 40.49 at 3B).Getting more from several models than from one · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A harness-evolution run where behavior-matched verification plus attribution gating yields no better held-out score than an equal-budget fixed or random verification batch, or where scaling fixed-split verification rollouts reliably closes the gap.