evolving-agent-harness-2-000-executable-tasks-disjoint-evaluation CLM-0331Evolving an agent harness on 2,000 executable tasks disjoint from the evaluation benchmarks, by contrasting successful and failed trajectories on the same task to find recurring deficiencies and evolving five harness modules (agent loop, tool use, observation management, context management, completion detection) separately before integrating them, raised DeepSeek-V4-Flash from 47.6 to 52.4 on TerminalBench 2.0 and 73.4 to 76.5 on SWE-Bench Verified, transferred across domains and to other backbones frozen (GLM-5.2 59.6 to 61.8, MiniMax-2.5 41.6 to 44.9), while evolving the whole harness at once or without modules fell below the unevolved baseline (46.4 and 44.2).
Capability: Improving its own scaffolding from failures
Observed on
DeepSeek-V4-Flash Preview as the evolving backbone, TerminalBench 2.0 and SWE-Bench Verified, 2026; evolution set curated from external sources disjoint from both benchmarks; trans.
Sources
- Validated by: Accuracy, Pass@3 and Pass-3 on TerminalBench 2.0 and SWE-Bench Verified for in-domain, cross-domain and cross-model transfer, plus an ablation comparing modular against non-modular and joint all-module evolution, where both non-modular variants scored below baseline. Developer-reported; one evolving backbone.. The ablation is the strong part: it shows the modular decomposition is load-bearing rather than incidental, and the benchmark-disjoint evolution set answers the contamination objection that hangs over SoL-Pi-style results. Gains are 3-5 points; [truncated]
- Secondary summary naming the three defects and fixes; its reply thread is where the four ratchet tests above were raised. Filed for provenance of the falsifier, not as independent evidence.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When an agent harness mines reusable rules and tools from its own labelled diagnosis trajectories, promoting every proposal that helps the cases it was mined from makes the harness worse over time than not evolving at all, and only admitting a proposal when it also fails to regress on a freshly resampled held-out case set keeps accuracy climbing across successive evolution rounds.Improving its own scaffolding from failures · reviewed by AI
- When an agent harness is evolved from execution traces under a fixed interaction budget, verifying each candidate edit on tasks chosen for the behavior it targets and accepting it only when the gain is attributable to that behavior beats scoring every candidate on a shared fixed, random or coreset task split, and larger verification budgets do not compensate: fixed-split methods using up to 24x the rollouts fell below the unmodified harness in half of their harness-benchmark pairs.Improving its own scaffolding from failures · reviewed by AI
- A large share of an agent benchmark score is attributable to the harness around the model rather than the model itself: holding the model fixed and changing only guides, sensors, retries and state management is reported to move the same model from 30.9% to 74.6% on GAIA and 30th to 5th on Terminal Bench — so a score reported for "a model" on an agentic benchmark is a score for a model-plus-harness, and comparisons across labs with different harnesses are not comparisons of models.Whether the measurement made the finding · reviewed by AI
- For LLM-driven heuristic discovery on combinatorial optimization tasks (TSP, CVRP, bi-objective and multi-component variants), letting an agent decide when to read, edit, diagnose and evaluate code while persisting each evaluation outcome as a retrievable experience card reaches better final heuristic quality than fixed evolutionary pipelines (EoH, ReEvo, FunSearch, MCTS-AHD, MEoH, MOTIF) at a fraction of their evaluation budget, and re-running the same task warm-started from the previous run's best code improves further round after round under a fixed budget.Fixing its own mistakes · reviewed by AI
- When a model must anticipate, without execution feedback, what a change to its own runtime harness components will do, naming which components a lifecycle change can affect stays near ceiling as the number of interacting effects grows from 2 to 32, while predicting the resulting state, reasoning across teardown orders, and choosing minimal reconfigurations degrade sharply (e.g. GPT-5.6 Luna's reachable-condition Jaccard 91.7% to 14.1%, executed reconfiguration success 62.5% to 25.0%).Improving its own scaffolding from failures · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Replication where whole-harness evolution matches modular evolution given equal evolution budget, or where the evolved harness fails to transfer to a backbone from a different family than those tested. Added 2026-09-21 after the DAIR.AI thread: an ablation in which the improver can versus cannot modify the evaluator it is judged by (its own tests, trace-reading prompts, or deficiency labels); if the gains shrink when that door is closed, the loop was fitting its judge. Four tests for a real ratchet raised in that thread, of which the paper meets three: (1) freeze the harness before evaluation, met by the frozen cross-model transfer; (2) forbid the improver from editing its own tests or notes, not addressed, the deficiency-finding step is itself a model reading traces with no stated separation from the evaluator; (3) require observability of failed paths, met by the contrastive success/failure traces; (4) promote only changes that survive a new session or disjoint task set, met by the benchmark-disjoint evolution set.