self-evolving-llm-agents-admit-new-skill-into-persistent-skill CLM-0635For self-evolving LLM agents that admit a new skill into a persistent skill bank whenever aggregate outcome reward rises, the accepted skills are collectively net-negative and the gate cannot do better in principle, because the bank-level gain is one scalar that leaves every incumbent skill's contribution unconstrained; scoring sampled coalition knockouts at the gate, and masking transferred skills whose removal improves retrieval quality on unlabeled target queries, recovers the missing composition and domain dependence.
Capability: Improving its own scaffolding from failures
Observed on
Skill-based self-evolving memory agents (MemSkill with a Llama-3.3-70B skill-extraction policy) on LoCoMo, LongMemEval, HotpotQA-50 and ALFWorld; three seeds; skills are textual me.
Sources
- Measured on one base agent framework and four benchmarks, three seeds, with a matched protocol differing only in the gate. The net-negative figure rests on 12 candidate proposals and a 50-item HotpotQA subset with a 20-query unlabeled probe, so the magnitudes are thin; the non-identifiability argument is analytic and does not depend on sample size. The design does not separate the benefit of rejecting more skills from the benefit of coalition-aware ranking per se, since the coalition gate simply accepts far fewer candidates.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When an agent harness mines reusable rules and tools from its own labelled diagnosis trajectories, promoting every proposal that helps the cases it was mined from makes the harness worse over time than not evolving at all, and only admitting a proposal when it also fails to regress on a freshly resampled held-out case set keeps accuracy climbing across successive evolution rounds.Improving its own scaffolding from failures · reviewed by AI
- For self-evolving agents that accumulate reusable skills in a bank and admit a new skill whenever aggregate task reward rises, the accepted skills can be collectively net-negative, because a bank-level reward change cannot identify which skill contributed and a skill's contribution depends on the other skills present and on the deployment domain.Improving its own scaffolding from failures · reviewed by AI
- For a frozen-weight video agent on open-ended surgical video QA, mining candidate retrieval skills from the agent's own low-scoring traces and keeping each only when it raises a four-dimensional validation reward on ~40 held-out questions beats installing a hand-written general 'looking discipline' skill, and the single distilled temporal skill (re-read the window adjacent to a localized event for before/after questions) carries most of the gain.Improving its own scaffolding from failures · reviewed by AI
- For LLM agents that score their own actions during exploration in text-based games and then reuse that experience, the accuracy of those self-scores against the environment verifier does not predict later gains on held-out stricter configurations: block-level correlations between judgment agreement or calibration and the next checkpoint's score change sit near zero, so recognizing a good action is not the bottleneck — converting it into a transferable policy is.Improving its own scaffolding from failures · reviewed by AI
- Evolving an agent harness on 2,000 executable tasks disjoint from the evaluation benchmarks, by contrasting successful and failed trajectories on the same task to find recurring deficiencies and evolving five harness modules (agent loop, tool use, observation management, context management, completion detection) separately before integrating them, raised DeepSeek-V4-Flash from 47.6 to 52.4 on TerminalBench 2.0 and 73.4 to 76.5 on SWE-Bench Verified, transferred across domains and to other backbones frozen (GLM-5.2 59.6 to 61.8, MiniMax-2.5 41.6 to 44.9), while evolving the whole harness at once or without modules fell below the unevolved baseline (46.4 and 44.2).Improving its own scaffolding from failures · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A matched-protocol run where the outcome-only gate's admitted skills are collectively positive when held-K-controlled, or where coalition-knockout gating and target-side retrieval masking fail to beat the outcome gate across banks and seeds. Drafted stance toward self-evolving-agents-accumulate-reusable-skills-bank-admit-new-skill: supports -- The K-controlled ablation measures that every skill the aggregate-reward gate admitted was individually accepted yet the set was collectively net-negative, and a non-identifiability theorem gives the reason.