distilling-lessons-from-failures-as-well-as-successes-beats-raw-trajectory-memory CLM-0516observationsingle paperReviewed by AI
On web-browsing (WebArena, Mind2Web) and software-engineering (SWE-Bench-Verified) benchmarks, an agent that distills reusable strategies from both its successful and its failed attempts, judged by an LLM without ground truth, and retrieves the relevant ones on later tasks, outperforms memories that store raw trajectories or only successful routines — up to 20% relative improvement in success and up to 16% fewer interaction steps — with the weights unchanged; including the failures is part of the gain, not noise.
Evidence for: Practice makes perfect (narrows)
Capability: Improving its own scaffolding from failures
Observed on
2025, Gemini 2.5 and Claude 3.7 agent backbones on WebArena, Mind2Web, SWE-Bench-Verified. Autonomous agent.
Applies when
- Effect on the capability:
- helps
- Helps most:
- independent
- Cost:
- moderate
- Fails when:
- The agent cannot judge its own outcomes even roughly: the gain held for judges between 70% and 90% accurate and was best with ground truth, and the paper does not test tasks where outcomes are hard to judge at all.
Sources
- Table 1 for the headline gains and step counts; the ablation that adds failed trajectories for the failure contribution. "Up to" figures are the best case across settings, not the typical one.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For LLM agents on long-horizon web penetration testing, holding the backbone model fixed and moving global state out of the context window onto a persistent graph of verified facts and fact-conditioned intent edges raises task success most on the hardest tasks, where success depends on facts obtained dozens of steps earlier (88.2% overall and 75.0% hard, vs 44.1% and 25.0% for the strongest shared-plain-text-memory baseline).Completing tasks that take many steps or many hours · reviewed by AI
- For LLM web agents on cluttered pages containing many concurrent interface components, success falls sharply as component count grows, and having the agent first emit a localized sub-goal plus a task-relevant sub-observation before choosing an action recovers much of the loss, because self-assessment of the action is more reliable in the reduced context.Using the tools it is given · reviewed by AI
- When a model gets a real environment signal after acting — a test result, a tool error, a task-success indicator — reflecting on that signal in words and retrying substantially improves success rates on multi-step coding and decision-making tasks over a single attempt.Fixing its own mistakes
- For single-agent procedural tasks where the relevant state fits a fixed schema known in advance, replacing the appended conversation transcript with a structured execution state that the model rewrites each step — discarding the reasoning trace once the state patch is validated — holds accuracy flat as the horizon grows while history-appending runtimes degrade, and the gain comes from the structure rather than the shorter prompt.Completing tasks that take many steps or many hours · reviewed by AI
- In multi-agent LLM pipelines over reasoning and code tasks with a pool of 3B-32B open-weight backbones, conditioning each per-step routing decision on a learned filtered memory rather than the full accumulated trajectory leaves accuracy about the same while cutting inference cost by roughly 40-50%, so most of the accumulated history is cost without accuracy value.Getting more from several models than from one · reviewed by AI
Notes
Weights frozen throughout, so the improvement lives in the memory, not the model: evidence for the narrows verdict on practice makes perfect. The paper's test-time scaling extension (MaTTS) is not filed separately.