arxiv-2509-25140 SRC-0580 · paperReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T. Le
Created: 2025-09-29 · Ingested: 2026-10-19
https://arxiv.org/abs/2509.25140(opens in a new tab)With the growing adoption of large language model agents in persistent real-world roles, they naturally encounter continuous streams of tasks. A key limitation, however, is their failure to learn from the accumulated interaction history, forcing them to discard valuable insights and repeat past errors. We propose ReasoningBank, a novel memory framework that distills generalizable reasoning strategies from an agent's self-judged successful and failed experiences. At test time, an agent retrieves relevant memories from ReasoningBank to inform its interaction and then integrates new learnings back, enabling it to become more capable over time. Building on this powerful experience learner, we further introduce memory-aware test-time scaling (MaTTS), which accelerates and diversifies this learning process by scaling up the agent's interaction experience. By allocating more compute to …
In brief
Storing distilled reasoning strategies from an agent's own failures, not just its successes, is what makes test-time memory pay off, and it also makes test-time scaling work better. ReasoningBank extracts titled, described, content-bearing strategy items from both successful and failed trajectories, labeled by an LLM-as-a-judge with no ground truth, and retrieves them into the system prompt for later tasks.
Testing covers WebArena (684 tasks across 5 subsets), Mind2Web cross-task/website/domain, and SWE-Bench-Verified, on Gemini-2.5-flash, Gemini-2.5-pro, and Claude-3.7-sonnet, against No Memory, trajectory memory (Synapse), and workflow memory (AWM).
Overall WebArena success rate rises +8.3, +7.2, and +4.6 over memory-free agents; SWE-Bench-Verified resolve rate goes 34.2 to 38.8 (flash) and 54.0 to 57.4 (pro). On WebArena-Shopping, adding failure traces helps ReasoningBank (46.5 to 49.7) but hurts AWM (44.4 to 42.2). Memory-aware scaling at k=5 reaches 55.1 parallel vs 52.4 for vanilla TTS and 42.2 without memory.
Effects are measured with ablations that isolate the pieces: memory-free scaling, scaling without aggregation, success-only vs success-plus-failure memory, and a judge-noise simulation showing stability from 70%-90% judge accuracy (measured baseline 72.7%).
Scaling curves and most analyses come from 1 subset (WebArena-Shopping) with 1 model, and MaTTS was not run on Mind2Web or SWE-Bench. Compute cost per task under scaling is deferred to an appendix, so equal-budget comparison is not in the main results.
If you build streaming agents, extract failure lessons rather than archiving winning trajectories.
Written from the abstract by claude-opus-5 on 2026-10-24, with every figure checked against it. Not a substitute for the paper.
Referenced by
Claims in this catalog that draw on this source, and whether as support or counterpoint.
- supportsOn web-browsing (WebArena, Mind2Web) and software-engineering (SWE-Bench-Verified) benchmarks, an agent that distills reusable strategies from both its successful and its failed attempts, judged by an LLM without ground truth, and retrieves the relevant ones on later tasks, outperforms memories that store raw trajectories or only successful routines — up to 20% relative improvement in success and up to 16% fewer interaction steps — with the weights unchanged; including the failures is part of the gain, not noise.
- supportsWhen an agent labels its own past attempts as successes or failures to build an experience memory, the labels can be substantially wrong without much cost: the LLM judge used was 72.7% accurate against ground truth on WebArena-Shopping, and simulating judges anywhere from 70% to 90% accurate gave similar success rates, with ground-truth labels best — so for distilling lessons, a rough self-judgment is close to enough, although a perfect one still helps.