arxiv-2609-08832 SRC-0277 · paper

Closing the Consistency Gap: Self-Evolving Agents That Learn to Stay on Course

Evelyn Duesterwald, Benjamin Elder, Lilian Ngweta, Shashanka Ubaru, Malgorzata Zimon

Created: 2026-09-08 · Ingested: 2026-09-19

Large language model (LLM)-powered agents can be accurate on average yet unreliable in production, a discrepancy that has been observed but remains largely unaddressed. When given the same task five times, a ReAct agent on the AppWorld benchmark using GPT-4.1 succeeds in all five runs only 53% of the time, even though its per-run pass rate averages 77%. We call this 24-point shortfall the consistency gap, and we argue that addressing it is a precondition for trustworthy AI agent deployment. We present a self-evolving agent framework that reduces this gap by identifying unstable, low-consistenc

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsFor ReAct agents on AppWorld, resampling each recorded inference point offline (N=30, temperature 0.5) to find steps whose responses vary across samples, then writing short natural-language guidelines targeting those steps into retrievable episodic memory, raises the fraction of tasks passing all five repeat runs by +16 points same-task and +13 points similar-task with GPT-4.1 (+6 and +8.7 with GPT-OSS-120B) without lowering mean per-run accuracy.