4b-8b-llm-agents-trained-plain-group-based-rl-grpo-emit CLM-0528When 4B-8B LLM agents are trained with plain group-based RL (GRPO) to emit variable-length action chunks in text environments, the policy degenerates — collapsing to one action per round on one backbone and over-committing to 5-6-action chunks with lower success on another — because a sparse terminal reward carries no information about where chunk boundaries belong; supplying boundary targets from subskills segmented out of successful trajectories yields an intermediate 3-4-action regime with both higher success and fewer decision rounds.
Capability: Completing tasks that take many steps or many hours
Observed on
Text-grounded interactive environments (ALFWorld, ScienceWorld) with binary terminal rewards, Qwen3-4B and Llama-3.1-8B-Instruct, max chunk length 6; not tested in stochastic or sa.
Sources
- Validated by: Success rate and average LLM decision rounds on ALFWorld and ScienceWorld seen/unseen splits against zero-shot, ReAct, Reflexion, RLOO, GRPO, GiGPO and a variable-length multi-action GRPO under a matched rollout budget: ALFWorld 96.1/94.5 vs 89.1/83.6 for GiGPO at 5.0/5.2 rounds; ScienceWorld 67.2/61.7 vs 35.9/34.4; ablations removing skills (86.7/88.3) or chunk-aware advantage (90.6/89.1) each lose part of the gain.. Measured on two benchmarks, two backbones, seen and unseen splits, with a matched rollout budget and component ablations; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When training an LLM with policy-gradient RL on tasks that decompose into many dependent sub-problems, rewarding each trajectory segment by how many reference reasoning points it newly reaches beats a sparse final-answer reward by a margin that widens with task horizon, and still learns on problem sets where the base policy never samples a success and sparse reward gives no gradient at all.Completing tasks that take many steps or many hours · reviewed by AI
- When training 4B-8B Qwen3 agents with group-based RL (GRPO, DAPO, GSPO) on sparse-reward long-horizon tasks, injecting hints into the environment's feedback — procedural action guidance in early episode steps and early training, switching to enriched state observations in later steps and later training — raises success on the unmodified environments by a few points on average, while lowering performance on tasks that require questioning whether the request is answerable at all.Completing tasks that take many steps or many hours · reviewed by AI
- For LLM agents that must compose and then execute domain tool workflows, training separate adapters for the two roles splits the gains cleanly: supervised fine-tuning of the planner on reference tool sequences mainly improves workflow-order fidelity, while node-level RL on argument-match rewards mainly improves tool-argument grounding and final-answer quality, and combining both beats either on a 9B backbone.Planning a sequence of actions before acting · reviewed by AI
- For 1.5B-7B Qwen2.5 backbones trained with RL on multi-hop retrieval QA, giving the policy reversible memory operations (summary, backtrack over a stack) with per-action process rewards raises F1 over trajectory-reward agentic-RAG baselines, and removing the two memory actions costs more than removing the planning action.Carrying a chain of inference through to its conclusion · reviewed by AI
- For LLM web agents on cluttered pages containing many concurrent interface components, success falls sharply as component count grows, and having the agent first emit a localized sub-goal plus a task-relevant sub-observation before choosing an action recovers much of the loss, because self-assessment of the action is more reliable in the reduced context.Using the tools it is given · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Variable-length-chunk RL with only a sparse terminal reward reaching comparable success and chunk granularity to skill-boundary-supervised training on the same environments and backbones, or the reported gain surviving when the skill-derived boundaries are replaced with arbitrary fixed-length segmentation.