multi-turn-agentic-rl-sparse-outcome-text-environments-alfworld-webshop-qwen2 CLM-0102For multi-turn agentic RL on sparse-outcome text environments (ALFWorld, WebShop) with Qwen2.5-1.5B/7B-Instruct, reading state and action values from two reserved vocabulary logits at the causal boundaries of the policy's own backbone, and assigning credit with turn-level GAE plus batch normalization, raises task success over PPO with a separate critic and over GRPO while sampling only one rollout per task.
Capability: Completing tasks that take many steps or many hours
Observed on
Two text environments, two Qwen2.5-Instruct sizes, three seeds, 150 training steps, sparse terminal rewards; aggregate gains only — not best in every task category, and recent grou.
Sources
- Measured success rates on ALFWorld (six categories plus aggregate) and WebShop score/success, mean and std over three seeds, against PPO, GRPO, RLOO, EMPG, GiGPO and prompting baselines; most baseline numbers are taken from prior work rather than rerun. Runtime breakdown shows 451.2s to 301.4s per iteration. No matched-compute comparison against GRPO with equal total rollouts, and the ablation contributions are not quantified in the given text.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For a small agent policy (Qwen3.5-4B) repeatedly deployed on the same task family with environment-level feedback, distilling only the trajectories whose textual-state guidance is newly or persistently useful relative to state-free execution raises the state-free policy's accuracy more than distilling all state-conditioned trajectories, and the resulting policy keeps improving over alternating rounds of state editing and consolidation.Improving its own scaffolding from failures · reviewed by AI
- When training 4B-8B Qwen3 agents with group-based RL (GRPO, DAPO, GSPO) on sparse-reward long-horizon tasks, injecting hints into the environment's feedback — procedural action guidance in early episode steps and early training, switching to enriched state observations in later steps and later training — raises success on the unmodified environments by a few points on average, while lowering performance on tasks that require questioning whether the request is answerable at all.Completing tasks that take many steps or many hours · reviewed by AI
- For multi-hop factual QA with Qwen2.5-7B/Qwen3-8B given a deliberately insufficient retrieved triple subgraph, training a policy with RL to insert bridging triples that let a separate frozen answerer reach the gold answer raises answer accuracy over standard GRPO and over prompting methods like Self-Ask and RECITE, and the improvement carries over to direct chain-of-thought answering with no triples inserted at test time.Stating false facts confidently · reviewed by AI
- When RL-training a central-agent multi-agent harness with 1.5B-3B Qwen2.5 backbones on multi-hop QA, optimizing only the coordinating central policy and leaving sub-agents frozen scores higher than jointly training central and sub-agent policies (average F1 47.79 vs 45.36, accuracy 42.88 vs 40.49 at 3B).Getting more from several models than from one · reviewed by AI
- For 1.5B-7B Qwen2.5 backbones trained with RL on multi-hop retrieval QA, giving the policy reversible memory operations (summary, backtrack over a stack) with per-action process rewards raises F1 over trajectory-reward agentic-RAG baselines, and removing the two memory actions costs more than removing the planning action.Carrying a chain of inference through to its conclusion · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Training the same method on other multi-turn environments or model families and finding it does not beat a tuned PPO/GRPO baseline at equal compute, or finding the gain comes from the turn-level advantage normalization alone rather than the shared value readout. Proposed technique, not catalogued: Shared-backbone value readout at causal boundaries with turn-level GAE.