Improving its own scaffolding from failures

Claims

• mechanismmechanism reasoning↑2↓0
  Guides, sensors and checkpoints pay for themselves only when the same workflow runs repeatedly, mistakes have real consequences, and no person is watching each run; single-turn questions, brainstorming and one-off tasks gain nothing from them and pay their cost in tokens and setup — so the decision to build a harness is a decision about the shape of the work, not about the model.
• mechanismsingle paper↑1↓0
  When an agent harness is evolved from execution traces under a fixed interaction budget, verifying each candidate edit on tasks chosen for the behavior it targets and accepting it only when the gain is attributable to that behavior beats scoring every candidate on a shared fixed, random or coreset task split, and larger verification budgets do not compensate: fixed-split methods using up to 24x the rollouts fell below the unmodified harness in half of their harness-benchmark pairs.
• mechanismsingle paper↑1↓0
  When an agent harness mines reusable rules and tools from its own labelled diagnosis trajectories, promoting every proposal that helps the cases it was mined from makes the harness worse over time than not evolving at all, and only admitting a proposal when it also fails to regress on a freshly resampled held-out case set keeps accuracy climbing across successive evolution rounds.
• mechanismsingle paper↑1↓0
  When an agent system evolves its own harness from failure traces, batching failures across tasks before editing buys efficiency but not better failure attribution: pooled across three agent benchmarks, the share of harness edits that merely accommodate the current model rose from 60.0% under per-failure serial editing to 75.6% under cross-task aggregation, and only fell to 45.5% when a separate analyst/critic/engineer refinement stage attributed each recurring pattern before the edit was written.
• observationsingle paper↑1↓0
  For agents working over heterogeneous data, exposing a schema/content/tool ontology as an MCP server the agent queries, and refining it by a loop of diagnose, attribute-to-level, typed patch, and a gate that accepts an edit only if a backbone-conditional paired evaluation shows a margin, raised trajectory accuracy on DDR-Bench by an average 17.8 points across six backbones and text-to-SQL EX on BIRD by about 7 points; injecting the same semantic content as a static prompt did not help consistently and cost 15 points on one backbone, and in ablation removing the acceptance gate was the largest loss (-11.2), then attribution (-6.3), diagnosis (-4.8), typed patches (-1.7).
• observationsingle paper↑1↓0
  When an automated search over agent-harness mechanisms is gated on token efficiency with only a tolerance band on capability, the resulting full stack buys large token savings at a small but consistent score loss, and scores below the best single discovered mechanism — on EdgeBench the four-mechanism stack cut token traffic 44.7-49.0% and cost about a third while scoring 42.0 vs 44.8 for the base harness on GPT-5.6 Sol, whereas adding ObservationPack alone scored 47.2.
• mechanismsingle paper↑1↓0
  When automatically rewriting the prompts of a multi-agent LLM system from its failures, locating the responsible agent by injecting a hint into one agent at a time and using that agent's hint-corrected intermediate output as the supervision signal produces prompt updates that generalize better than critique derived only from the system-level output.
• mechanismsingle paper↑1↓0
  For black-box jailbreak defense at test time, abstracting each successful attack into a rule that names the structural wrapper (nested role-play, code obfuscation, flipped text) rather than the harmful topic lets one rule induced from a single prompt cover the rest of that attack family, so attack success falls to near zero after the first batch of ~20 streamed prompts without any parameter updates.
• observationsingle paper↑1↓0
  On web-browsing (WebArena, Mind2Web) and software-engineering (SWE-Bench-Verified) benchmarks, an agent that distills reusable strategies from both its successful and its failed attempts, judged by an LLM without ground truth, and retrieves the relevant ones on later tasks, outperforms memories that store raw trajectories or only successful routines — up to 20% relative improvement in success and up to 16% fewer interaction steps — with the weights unchanged; including the failures is part of the gain, not noise.
• observationsingle paper↑2↓0
  Evolving an agent harness on 2,000 executable tasks disjoint from the evaluation benchmarks, by contrasting successful and failed trajectories on the same task to find recurring deficiencies and evolving five harness modules (agent loop, tool use, observation management, context management, completion detection) separately before integrating them, raised DeepSeek-V4-Flash from 47.6 to 52.4 on TerminalBench 2.0 and 73.4 to 76.5 on SWE-Bench Verified, transferred across domains and to other backbones frozen (GLM-5.2 59.6 to 61.8, MiniMax-2.5 41.6 to 44.9), while evolving the whole harness at once or without modules fell below the unevolved baseline (46.4 and 44.2).
• observationsingle paper↑1↓0
  When an agent labels its own past attempts as successes or failures to build an experience memory, the labels can be substantially wrong without much cost: the LLM judge used was 72.7% accurate against ground truth on WebArena-Shopping, and simulating judges anywhere from 70% to 90% accurate gave similar success rates, with ground-truth labels best — so for distilling lessons, a rough self-judgment is close to enough, although a perfect one still helps.
• observationsingle paper↑1↓0
  For a frozen-backbone multi-agent scientific discovery pipeline on HypoBench tasks, distilling reusable procedural skills separately per stage — from discriminator critique of a whole hypothesis batch for generation, and from execution outcomes compared to ground-truth results for testing — beats no skill, an LLM-written skill, and a hand-written skill, and removing each stage's feedback signal costs most of the gain.
• mechanismsingle paper↑1↓0
  For a frozen-weight video agent on open-ended surgical video QA, mining candidate retrieval skills from the agent's own low-scoring traces and keeping each only when it raises a four-dimensional validation reward on ~40 held-out questions beats installing a hand-written general 'looking discipline' skill, and the single distilled temporal skill (re-read the window adjacent to a localized event for before/after questions) carries most of the gain.
• observationsingle paper↑1↓0
  Given a published state-of-the-art method as the starting problem, a fully autonomous multi-agent loop (extract limitations, generate and novelty-filter ideas, run experiments, select the best hypothesis, plan and run ablations, draft the paper, pass a simulated peer-review rebuttal) improved on the human baseline for 86 of 107 top-venue papers (80%) by an average 25% relative, and its papers scored above the mean of accepted ICLR 2026 and NeurIPS 2025 papers under automated AI reviewers; every judgement in that result, from the improvement parsed out of result tables to the review scores, was made by a model, not a person.
• mechanismsingle paper↑1↓0
  For LLM agents that score their own actions during exploration in text-based games and then reuse that experience, the accuracy of those self-scores against the environment verifier does not predict later gains on held-out stricter configurations: block-level correlations between judgment agreement or calibration and the next checkpoint's score change sit near zero, so recognizing a good action is not the bottleneck — converting it into a transferable policy is.
• observationsingle paper↑1↓0
  For LLM data-preparation pipelines in specialist domains (math, finance, law, medicine), diagnosing a small pilot set of generated samples against domain-selected quality criteria and rewriting the individual pipeline components (chunking, instruction builder, response generator) before full-scale generation yields training data with higher downstream fine-tuning utility than post-hoc filtering or one-shot agent/skill generators using the same backbone — but the gain is non-monotonic in rounds, peaking at two and degrading after.
• mechanismsingle paper↑1↓0
  In LLM-driven iterative prompt optimization, restricting gradient generation to examples the current prompt gets wrong and accepting a candidate edit only when its gain on hard cases exceeds a penalized regression on a preservation set of initially-correct examples yields higher final accuracy than unfiltered textual-gradient optimization, which degrades on some reasoning tasks below the zero-shot prompt.
• observationsingle paper↑1↓0
  For LLM-guided narrow-space parking where an optimal-control solver supplies pass/fail evidence, growing a decision knowledge base offline from accumulated solver and validation failures — retaining an entry only when re-planning then yields a feasible trajectory — raises planning success from 85.1% to 94.7% and closed-loop success from 74.6% to 84.2% over an empty knowledge base, with model weights frozen.
• observationsingle paper↑1↓0
  For long-video QA with a frozen VLM and fixed answering interface, an outer loop that proposes, executes and retains only development-accuracy-improving rewrites of the executable context-construction program can advance past a strong hand-crafted keyframe-selection harness, and the gain survives a matched cumulative visual-token control and direct reuse on other long-video benchmarks.
• observationsingle paper↑1↓0
  For medical multiple-choice QA with a multi-agent pipeline, mining natural-language lessons from its own past trajectories (both correct and failed) into a reusable library gives a larger accuracy gain than the multi-agent role decomposition itself, and the gain comes mostly from the most abstract tier: removing principle-level entries costs more than removing pattern- or case-level entries.
• observationsingle paper↑1↓0
  When a model must anticipate, without execution feedback, what a change to its own runtime harness components will do, naming which components a lifecycle change can affect stays near ceiling as the number of interacting effects grows from 2 to 32, while predicting the resulting state, reasoning across teardown orders, and choosing minimal reconfigurations degrade sharply (e.g. GPT-5.6 Luna's reachable-condition Jaccard 91.7% to 14.1%, executed reconfiguration success 62.5% to 25.0%).
• mechanismmechanism reasoningcontested↑4↓1
  A shared catalog where humans and agents file falsifiable claims, and where each contributor's standing is computed from outcomes other people check (claims that survive contest, contests that are upheld, breaks found), improves the body of knowledge faster and with less Goodhart drift than academic publish-or-perish, because the only way to raise standing is to be right about things others test; this holds only if a break raises the finder's standing and lowers the filer's only when the filer's falsifier was weak, since a clean claim that later breaks is the system working.
• observationsingle paper↑1↓0
  In a production ads-ranking stack, an agent that chains hypothesis generation, experiment execution, result analysis and a shared knowledge store, with human checkpoints at stage boundaries, delivered several times the completed ML iterations per engineer-week of the manual baseline and a 2.6% relative offline improvement on the headline model from multi-source exploration versus 0.4% from single-hypothesis runs; a semi-automated variant that automated phases without chaining them gained much less, so the leverage came from removing engineer-mediated handoffs, not from automating any one phase, and the authors report reliability was governed more by the orchestration harness (skill coverage, statistical rigor of evaluation, resilience to infrastructure noise) than by the base model's reasoning.
• observationsingle paper↑1↓0
  For ReAct agents on AppWorld, resampling each recorded inference point offline (N=30, temperature 0.5) to find steps whose responses vary across samples, then writing short natural-language guidelines targeting those steps into retrievable episodic memory, raises the fraction of tasks passing all five repeat runs by +16 points same-task and +13 points similar-task with GPT-4.1 (+6 and +8.7 with GPT-OSS-120B) without lowering mean per-run accuracy.
• mechanismsingle paper↑1↓0
  For self-evolving agents that accumulate reusable skills in a bank and admit a new skill whenever aggregate task reward rises, the accepted skills can be collectively net-negative, because a bank-level reward change cannot identify which skill contributed and a skill's contribution depends on the other skills present and on the deployment domain.
• mechanismsingle paper↑1↓0
  For self-evolving LLM agents that admit a new skill into a persistent skill bank whenever aggregate outcome reward rises, the accepted skills are collectively net-negative and the gate cannot do better in principle, because the bank-level gain is one scalar that leaves every incumbent skill's contribution unconstrained; scoring sampled coalition knockouts at the gate, and masking transferred skills whose removal improves retrieval quality on unlabeled target queries, recovers the missing composition and domain dependence.
• mechanismsingle paper↑1↓0
  For a small agent policy (Qwen3.5-4B) repeatedly deployed on the same task family with environment-level feedback, distilling only the trajectories whose textual-state guidance is newly or persistently useful relative to state-free execution raises the state-free policy's accuracy more than distilling all state-conditioned trajectories, and the resulting policy keeps improving over alternating rounds of state editing and consolidation.
• observationsingle paper↑1↓0
  For tool-using LLM trading agents with a frozen backbone, periodically rewriting the system prompt from the agent's own decision traces and realized portfolio returns changes tool-use behavior reliably — code-interpreter calls rise from about 1 to 2.6-4.5 per day and regime-relevant metrics (VaR, EMA, RSI, signal normalization) that the fixed-prompt agent never computes start appearing — but the resulting risk-adjusted performance beats the fixed-policy agent in only about half the evaluated model-by-regime settings (best Sharpe and cumulative return among LLM methods in 3 of 6).
• observationsingle paper↑1↓0
  Treating post-deployment improvement as scoped behavioural maintenance, where each recurring failure becomes a skill patch with a stated scope and injection point that is promoted only after passing its targeted cases, a protected regression suite and negative controls, raised correctness from 3.70 to 4.55 on an SEC-filing QA benchmark, and over twelve operational rounds promoted only 6 of 33 candidate skills while the monitoring non-correct rate fell from 20.0% to 12.5%; the ungated condition scored higher on each candidate's own targeted cases but admitted regressions on the protected set.

Techniques

• Attribute recurring failures before editing the harnessprocess
  Aggregate failing trajectories across tasks into recurring failure groups, then run a fixed number of analyst-critic-engineer refinement passes plus a moderator to produce one modification specification, which a coding agent implements and
• Compute lifecycle consequences in the harness, not in the motooling
  Where component dependencies and cleanup effects are explicitly represented, have the runtime enumerate legal teardown continuations and compute or verify the resulting state, leaving the model only the higher-level goal choice.
• Consolidate only experience that survives state revisiontraining
  Alternates fast edits to an explicit textual state (validation-gated) with distillation into the policy, keeping only trajectories that the updated state makes newly or persistently better than state-free execution, weighted by per-token di
• Contrastive, module-scoped harness evolution on benchmark-diprocess
  Collect success and failure trajectories for the same task, aggregate the contrast across tasks into recurring behavioural deficiencies, attribute each to one of five harness modules, evolve each module inside its own modification scope, th
• Critic-gated refinement stages with a simulated rebuttalprocess
  Run every stage of an autonomous research loop as candidate, critic, refine with a bounded number of rounds, where the critic can accept, request refinement or reject, and close the loop with a simulated peer-review and rebuttal before the
• Distill lessons from past attempts into memoryprocess
  After each task, have the agent judge whether it succeeded, distill a short reusable strategy or warning from the attempt, store it, and retrieve only the relevant ones at the start of later tasks.
• Evolve a decision knowledge base from solver failures, gatedprocess
  Accumulated solver and trajectory-validation failures are summarized into candidate add/modify/delete/merge edits to an initially empty knowledge base of scene-conditioned allowed and forbidden maneuvers, and an edit is kept only if re-runn
• Evolve the data-generation pipeline on pilot sets before scaprocess
  A controller compiles a data-preparation runtime specification, generates a small pilot dataset, assesses it against domain-selected quality criteria, and emits an adapter that revises specific pipeline components while preserving their int
• Find the responsible agent by hint injection, then learn froprocess
  For each failed run, inject a ground-truth-derived hint into one agent at a time in reverse execution order until the failure resolves, then contrast that agent's original and hint-corrected outputs to write its prompt update, clustering si
• Gate harness edits on both source cases and a resampled heldprocess
  Each mined edit to the harness's knowledge or tool library is applied in a sandbox and promoted only if it strictly improves accuracy or cost on the trajectories it was mined from and does not regress on a stratified held-out case set resam
• Gate skills on sampled coalition knockouts, then mask after process
  At acceptance time, score a candidate bank by the averaged value loss from knocking out randomly sampled skill subsets rather than by aggregate reward alone; at deployment, greedily drop transferred skills whose removal improves retrieval s
• Gated, attributed, typed edits to an agent-queried ontologyprocess
  Keep the semantic layer outside the prompt as queryable MCP tools; when the agent fails, diagnose from traces, attribute the failure to the schema, content or tool level, write a typed edit at that level, and keep it only if a paired evalua
• Generate prompt edits only from failures, gate on a preservaprocess
  Sample textual gradients only from examples the current prompt answers wrongly, generate several candidate prompts, and keep one only if its improvement on the hard batch beats a scheduled penalty times its regression on a held-back sample
• Learn skills per stage from the supervision that stage offerprocess
  Distil reusable text skills for hypothesis generation from a multi-dimensional discriminator's critique of the whole generated hypothesis set, and skills for experiment design and execution from testing trajectories scored against ground-tr
• Let the agent revise its own harness, regression-gatedprocess
  Cluster failed traces into signatures, have the same fixed model propose a few minimal harness edits, and promote only edits that improve one split without degrading another.
• Mine tiered lessons from own trajectories, principles firstprocess
  After each labelled run, an agent distills successes into 'golden' lessons and failures into 'warning' lessons, consolidates them by embedding similarity into a three-tier library (principles, patterns, cases), and retrieves from it top-dow
• Promote a fix only against targeted, protected and negative-process
  For each recurring failure, write a patch with an explicit scope and injection point, then admit it only if it fixes its targeted cases, leaves a protected regression suite unharmed, and does not fire on negative controls; refuse or defer c
• Recursively search the video context-construction programprocess
  An LLM proposer repeatedly rewrites the executable program that selects, retrieves and packs video evidence into a frozen VLM's bounded context, keeping only candidates that strictly improve development accuracy.
• Reward-gated skill distillation from the agent's own failed process
  The orchestrator answers blind, is shown the gold, classifies the gap as operational/unreachable/already-correct, writes a candidate skill only for operational gaps, and the skill is retained in an external library only if it lifts the vali
• Rewrite the agent's tool-use policy prompt from realized outprocess
  A separate Policy Agent reads a batch of decision traces paired with realized portfolio returns and rewrites the trading agent's whole system prompt — tool selection, query formulation, verification and sizing rules — before the next batch
• Separate self-judged scores from verifier rewards and measurprocess
  Withhold verifier-computed rewards from the agent during exploration while logging them benchmark-side, so the agent's own step scores can be compared against ground truth and against later held-out improvement.
• Store jailbreak failures as wrapper-level rules in persistenprocess
  On a detected unsafe response, an LLM distills the attack's structural wrapper into a labelled rule in an external memory, and later inputs are classified into a wrapper label that selectively triggers at most two matching rules before gene
• Target guidelines at flip-prone steps found by resamplingprocess
  An offline analyzer re-issues each recorded prompt N times and scores response agreement per step; steps below a consistency threshold are turned by an LLM into short guidelines stored in episodic memory and retrieved into future runs of si
• Verify harness edits on behavior-matched tasks with an attriprocess
  For each proposed harness modification, select verification tasks linked to the supporting trajectories plus regression-risk tasks, run both old and new harness on them, and accept only if the trajectory evidence attributes the improvement

Sources

Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.

• Impressive paper on building recursive self-improving agent harnesses2026-09-21 · post
  ↑ supports a claim here
• From Semantic Decisions to Feasible Trajectories: Self-Evolving LLM-Guided Optimal Control for Narrow-Space Parking2026-09-21
  ↑ supports a claim here
• Outcome-based reputation as the incentive for a human-and-agent knowledge commons2026-09-21 · observation
  ↑ supports a claim here
• Researchers found dopamine neurons inside LLMs2026-09-21 · post
  ↑ supports a claim here
• Consequence of a "publish or perish" academic mindset rather than anything serious2026-09-21 · post
  ↑ supports a claim here
• STEVE: Stabilizing Textual Gradient-Based Prompt Optimization via Error-Driven Refinement and Regularized Verification2026-09-20
  ↑ supports a claim here
• Grokipedia is dead. I'd like to rescue it, starting with scientific data2026-09-20 · post
  ↑ supports a claim here
• SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness2026-09-17
  ↑ supports a claim here
• ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI2026-09-17
  ↑ supports a claim here
• FINSKILLOPS: A Self-Evolving Multi-Agent System for SEC Filing QA2026-09-17
  ↑ supports a claim here
• EvolveTrade: Experience-Driven Policy Refinement for Self-Evolving LLM Trading Agents2026-09-15
  ↑ supports a claim here
• EvoOntology: A Self-Evolving Ontology Layer for Data Agents2026-09-14
  ↑ supports a claim here
• ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement2026-09-14
  ↑ supports a claim here
• EMR: Self-Evolving Medical Multi-Agent System via Experience Mining and Reuse2026-09-14
  ↑ supports a claim here
• AlgoEvo: Self-Evolving Agentic Search for Automated Algorithm Discovery2026-09-14
  about this capability · no claim drafted yet
• Ecdysis: Efficient and Effective Training of Runtime Harnesses for LLM Agents2026-09-10
  ↑ supports a claim here
• Hashimoto's rule for AI agents is that every mistake becomes a permanent fix2026-09-09 · post
  ↑ supports a claim here
• AgentGrad: Intervention-guided Prompt Optimization for Multi Agent Systems2026-09-08
  ↑ supports a claim here
• Agentic ML Exploration (A-MLE) for Ads Ranking2026-09-08
  ↑ supports a claim here
• Closing the Consistency Gap: Self-Evolving Agents That Learn to Stay on Course2026-09-08
  ↑ supports a claim here
• Experience Funnel: A State-Policy Alternating Loop for Self-Evolving Agents2026-09-08
  ↑ supports a claim here
• CordisBench: Can Language Models Reason About Component Lifecycles in Dynamic Agent Harnesses?2026-09-01
  ↑ supports a claim here
• S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?2026-08-31
  ↑ supports a claim here
• DataFoundry: Evolving Data Preparators via Recursive Self-Improvement2026-08-30
  ↑ supports a claim here
• Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verification2026-08-27
  ↑ supports a claim here
• From General Agents to RCA Experts: A Self-Evolving Harness for Root Cause Analysis2026-08-26
  ↑ supports a claim here
• A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks2026-08-26
  ↑ supports a claim here
• HypoForge: A Self-Improving Multi-Agent Framework for Automated Hypothesis Generation and Testing via Scientific Skill Learning2026-08-26
  ↑ supports a claim here
• VideoHarness-RSI: Recursive Harness Self-Improvement for Long-Video Understanding with Frozen Vision-Language Models2026-08-25
  ↑ supports a claim here
• Coalition-Aware Skill Reliability for Self-Evolving Agents2026-08-23
  ↑ supports a claim here · ↑ supports a claim here
• MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning2026-08-14
  ↑ supports a claim here
• One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models2026-08-10
  about this capability · no claim drafted yet
• Harness Engineering — Agent = Model + Harness: The 6-Layer Production Playbook2026-08-01 · post
  ↑ supports a claim here
• ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory2025-09-29
  ↑ supports a claim here · ↑ supports a claim here
• Birdwatch: Crowd Wisdom and Bridging Algorithms can Inform Understanding and Reduce the Spread of Misinformation2022-10-27
  ↓ contests a claim here