self-reflection-used-training-signal-long-horizon-agent-math-tasks-resetting CLM-0147When self-reflection is used as a training signal for long-horizon agent and math tasks, resetting to the original prompt with a compact reflection patch prepended beats accumulating reflections inside the ongoing trajectory, and overly long reflections (more than ~10 points) hurt compared with 2-5 bullet points.
Capability: Completing tasks that take many steps or many hours
Observed on
Qwen3 1.5B/8B/32B and Llama-3.1-8B-Instruct post-trained with reflection-conditioned on-policy distillation; verifiable-outcome tasks (AIME'24, WebShop, ALFWorld, SWE-Bench-Lite)..
Sources
- Single-paper ablation table: full SRPO 73.3 AIME'24 / 64.7 WebShop vs 66.3 / 52.8 for no state reset, 68.5 / 57.4 for appended reflection, 70.0 / 60.3 for verbose reflection. One ablation run per row, no variance reported for the ablations; only one model size ablated, so the mechanism claim is argued more than isolated.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In LLM multi-agent systems that reflect after failure, restricting reflection to the single agent responsible for the first incorrect step — located by a trained step-level correctness scorer and paired with a counterfactually generated corrected step — raises task success more than having every agent reflect on the trajectory, because reflections written by agents that behaved correctly enter their memory as misleading guidance for later tasks.Getting more from several models than from one · reviewed by AI
- For long-horizon LLM agents in environments where an erroneous action changes world state, restoring an earlier checkpoint and carrying forward a structured reflection memory that excludes branch-local state claims beats either intervention alone, because reflection alone cannot undo environmental damage and rollback alone re-explores the same failure.Completing tasks that take many steps or many hours · reviewed by AI
- For 8B instruction-tuned backbones, training a LoRA corrector to diagnose a typed, executable error operator (error type, step location, corrupted and repaired proposition) and apply its inverse as a localized edit raises accuracy over free-form self-revision baselines (Self-Refine, RISE, SCoRe) on math, code and commonsense tasks, and dropping either the operator-prediction loss or the adjacent-state reconstruction loss loses most of the gain.Fixing its own mistakes · reviewed by AI
- In open-weight instruction-tuned models, interventions that suppress caving to user pushback (DPO, SFT on chosen responses, or activation steering) also tend to reduce the model's rate of correcting a wrong answer when genuine supporting evidence is supplied, because the two answer-flip behaviors run on overlapping MLP neurons and attention heads with positively aligned steering directions.Telling the user what they want to hear · reviewed by AI
- When fine-tuning small open models (4B-8B Qwen3) for multi-turn tool calling, supervising only the recovery steps generated after the first point where the student's rollout leaves the teacher-supported state region — and zeroing the loss on the student's own valid prefix — beats full-trajectory imitation and outcome-reward RL on in-domain and held-out tool benchmarks, and the trained student's successful trajectories get shorter across iterations rather than longer.Using the tools it is given · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: An ablation under the same training setup where appending reflections iteratively matches or beats reset-with-memory, or where verbose reflections (>10 points) score at least as well as compact ones. Drafted stance toward adding-irrelevant-tokens-alone-degrades-reasoning-mechanism-by: supports -- Verbose reflections and iteratively appended reflection context both lowered scores relative to a compact prepended patch on a clean prompt, consistent with accumulated context degrading reasoning. Proposed technique, not catalogued: Reset-with-memory reflection patches as dense training signal.