training-llm-policy-gradient-rl-tasks-decompose-into-many-dependent CLM-0501When training an LLM with policy-gradient RL on tasks that decompose into many dependent sub-problems, rewarding each trajectory segment by how many reference reasoning points it newly reaches beats a sparse final-answer reward by a margin that widens with task horizon, and still learns on problem sets where the base policy never samples a success and sparse reward gives no gradient at all.
Capability: Completing tasks that take many steps or many hours
Observed on
Requires one reference solution per task and a way (symbolic check or LLM judge) to test whether a segment reached a reference reasoning point; shown on 1.7B-4B Qwen3 models on syn.
Sources
- Validated by: GRPO baseline vs trajectory-level and segment-level variants on Multi-Countdown, Matrix Manipulation, GSM-Infinite (success 0.415 vs 0.084 sparse at n=16; 0.191 vs 0.091 at n=24), Polaris pass@1/pass@8 at 16K eval, and a POPE-hard subset where base pass rate is ~0.004 at 8K; also a signal-to-noise theorem.. Measured on three synthetic environments plus two math datasets with small Qwen3 models; baselines include GRPO, SFT, Verifree, OPSD and a process-reward judge. Margins on Polaris are small and some test splits favor baselines. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When a calculator-using model is trained with reinforcement learning on verifiable final-answer rewards for an arithmetic search task (Countdown), the gain lands almost entirely at low k — pass@1 rises sharply while pass@16 moves little or falls — because the update can only reinforce correct trajectories the starting policy already samples, and prompt groups with no correct sample supply no gradient.Using the tools it is given · reviewed by AI
- For LLM-guided narrow-space parking where an optimal-control solver supplies pass/fail evidence, growing a decision knowledge base offline from accumulated solver and validation failures — retaining an entry only when re-planning then yields a feasible trajectory — raises planning success from 85.1% to 94.7% and closed-loop success from 74.6% to 84.2% over an empty knowledge base, with model weights frozen.Improving its own scaffolding from failures · reviewed by AI
- When 4B-8B LLM agents are trained with plain group-based RL (GRPO) to emit variable-length action chunks in text environments, the policy degenerates — collapsing to one action per round on one backbone and over-committing to 5-6-action chunks with lower success on another — because a sparse terminal reward carries no information about where chunk boundaries belong; supplying boundary targets from subskills segmented out of successful trajectories yields an intermediate 3-4-action regime with both higher success and fewer decision rounds.Completing tasks that take many steps or many hours · reviewed by AI
- For long-horizon embodied task planning where a symbolic model defines action preconditions and effects, correcting a violated plan by edits the action model itself implies (inserting the missing navigate or open) raises plan success more per LLM call than returning the violation to the LLM for replanning, and the gain persists for frontier models as well as compact ones.Planning a sequence of actions before acting · reviewed by AI
- For 1.5B-7B Qwen2.5 backbones trained with RL on multi-hop retrieval QA, giving the policy reversible memory operations (summary, backtrack over a stack) with per-action process rewards raises F1 over trajectory-reward agentic-RAG baselines, and removing the two memory actions costs more than removing the planning action.Carrying a chain of inference through to its conclusion · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: On long-horizon tasks, segment-level reasoning-point rewards match or trail sparse outcome rewards in final success rate, or the gap fails to grow with horizon, under matched compute and reference-trajectory access.