training-4b-8b-qwen3-agents-group-based-rl-grpo-dapo-gspo CLM-0499When training 4B-8B Qwen3 agents with group-based RL (GRPO, DAPO, GSPO) on sparse-reward long-horizon tasks, injecting hints into the environment's feedback — procedural action guidance in early episode steps and early training, switching to enriched state observations in later steps and later training — raises success on the unmodified environments by a few points on average, while lowering performance on tasks that require questioning whether the request is answerable at all.
Capability: Completing tasks that take many steps or many hours
Observed on
SciWorld and BFCL-v3 multi-turn, Qwen3-4B/8B, 200 RL steps, 0.5 enrichment probability, feedback held consistent within a rollout group; on AppWorld the method failed to escape zer.
Sources
- supportsEnvironments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon TasksValidated by: Success rate on unenriched standard environments, compared against training in standard environments, across 2 model scales x 3 RL algorithms x 2 benchmarks; average absolute gain 2.82 points, with a hybrid schedule beating all four single strategies in the pilot study; regressions of 3-7 points on BFCL Miss Param / Miss Func splits.. Measured on two benchmarks with three RL algorithms and two model sizes; gains are small (0.69 to 4.61 points average) and non-uniform, with several per-split drops. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For a small agent policy (Qwen3.5-4B) repeatedly deployed on the same task family with environment-level feedback, distilling only the trajectories whose textual-state guidance is newly or persistently useful relative to state-free execution raises the state-free policy's accuracy more than distilling all state-conditioned trajectories, and the resulting policy keeps improving over alternating rounds of state editing and consolidation.Improving its own scaffolding from failures · reviewed by AI
- For 1.5B-7B Qwen2.5 backbones trained with RL on multi-hop retrieval QA, giving the policy reversible memory operations (summary, backtrack over a stack) with per-action process rewards raises F1 over trajectory-reward agentic-RAG baselines, and removing the two memory actions costs more than removing the planning action.Carrying a chain of inference through to its conclusion · reviewed by AI
- When 4B-8B LLM agents are trained with plain group-based RL (GRPO) to emit variable-length action chunks in text environments, the policy degenerates — collapsing to one action per round on one backbone and over-committing to 5-6-action chunks with lower success on another — because a sparse terminal reward carries no information about where chunk boundaries belong; supplying boundary targets from subskills segmented out of successful trajectories yields an intermediate 3-4-action regime with both higher success and fewer decision rounds.Completing tasks that take many steps or many hours · reviewed by AI
- When fine-tuning a 14B model on knowledge-graph-derived multiple-choice QA for a single medical domain, attaching supporting triples from the same source text chunk to each target triple during supervision, and repairing residual one-hop failures before reinforcement learning, gives higher accuracy on held-out 3-, 4- and 5-hop questions than training on isolated triples or running RL from an unrepaired checkpoint.Carrying a chain of inference through to its conclusion · reviewed by AI
- In GRPO-style RL post-training of 7B multimodal reasoning models, composing each optimization group from rollouts stratified by length-normalized sequence surprisal and outcome (confident-correct anchors, confident-wrong negatives, low-confidence successes) within the same step keeps path entropy from collapsing and avoids the late-training performance decline GRPO and DAPO show after a few hundred steps.Retaining skills and safety after fine-tuning · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Training in the feedback-enriched environments yields no better or worse success on the standard environments than standard-environment RL at matched steps, or a fixed schedule (guidance throughout, or enrichment throughout) matches the early-guidance/late-enrichment switch.