agent-harness-mines-reusable-rules-tools-own-labelled-diagnosis CLM-0546When an agent harness mines reusable rules and tools from its own labelled diagnosis trajectories, promoting every proposal that helps the cases it was mined from makes the harness worse over time than not evolving at all, and only admitting a proposal when it also fails to regress on a freshly resampled held-out case set keeps accuracy climbing across successive evolution rounds.
Capability: Improving its own scaffolding from failures
Observed on
Root cause analysis over microservice telemetry, general coding agents (Codex, Claude Code) with four backbones, 12 continuous-diagnosis windows over two public benchmarks; effect .
Sources
- Validated by: Per-window top-1 accuracy over 12 continuous-diagnosis windows on OpenRCA and RCAEval across four backbones: full system final-window A@1 0.83 vs 0.43 for the non-evolving harness and 0.33 for the ungated variant (0.4 to 0.0 collapse on DeepSeek-V4); 37% of proposals rejected by the gate.. Measured ablation with three variants across four backbones and two benchmarks, with per-window curves rather than a single endpoint. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For self-evolving LLM agents that admit a new skill into a persistent skill bank whenever aggregate outcome reward rises, the accepted skills are collectively net-negative and the gate cannot do better in principle, because the bank-level gain is one scalar that leaves every incumbent skill's contribution unconstrained; scoring sampled coalition knockouts at the gate, and masking transferred skills whose removal improves retrieval quality on unlabeled target queries, recovers the missing composition and domain dependence.Improving its own scaffolding from failures · reviewed by AI
- Evolving an agent harness on 2,000 executable tasks disjoint from the evaluation benchmarks, by contrasting successful and failed trajectories on the same task to find recurring deficiencies and evolving five harness modules (agent loop, tool use, observation management, context management, completion detection) separately before integrating them, raised DeepSeek-V4-Flash from 47.6 to 52.4 on TerminalBench 2.0 and 73.4 to 76.5 on SWE-Bench Verified, transferred across domains and to other backbones frozen (GLM-5.2 59.6 to 61.8, MiniMax-2.5 41.6 to 44.9), while evolving the whole harness at once or without modules fell below the unevolved baseline (46.4 and 44.2).Improving its own scaffolding from failures · reviewed by AI
- For a frozen-weight video agent on open-ended surgical video QA, mining candidate retrieval skills from the agent's own low-scoring traces and keeping each only when it raises a four-dimensional validation reward on ~40 held-out questions beats installing a hand-written general 'looking discipline' skill, and the single distilled temporal skill (re-read the window adjacent to a localized event for before/after questions) carries most of the gain.Improving its own scaffolding from failures · reviewed by AI
- For self-evolving agents that accumulate reusable skills in a bank and admit a new skill whenever aggregate task reward rises, the accepted skills can be collectively net-negative, because a bank-level reward change cannot identify which skill contributed and a skill's contribution depends on the other skills present and on the deployment domain.Improving its own scaffolding from failures · reviewed by AI
- When an agent harness is evolved from execution traces under a fixed interaction budget, verifying each candidate edit on tasks chosen for the behavior it targets and accepting it only when the gain is attributable to that behavior beats scoring every candidate on a shared fixed, random or coreset task split, and larger verification budgets do not compensate: fixed-split methods using up to 24x the rollouts fell below the unmodified harness in half of their harness-benchmark pairs.Improving its own scaffolding from failures · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An ablation on comparable tasks where the ungated self-evolving harness matches or exceeds the dual-gated one across many evolution rounds, or where removing the held-out gate leaves accuracy at or above the static-harness baseline. Drafted stance toward self-evolving-agents-accumulate-reusable-skills-bank-admit-new-skill: supports -- Both find that admitting a skill because aggregate reward rose on the mined cases produces a net-negative accumulated library, and this paper measures the resulting decay directly.