4b-30b-qwen3-backbones-trained-self-play-rl-where-same-model CLM-0003For 4B-30B Qwen3 backbones trained with self-play RL where the same model writes executable multi-turn tool environments (simulated tool schemas, backend state, per-instruction state checks) and then solves them, multi-turn function-calling scores rise, and the gain is largest on the benchmark whose task structure most closely mirrors the generated environments (ACEBench-Agent +13.9 at 30B-A3B) and smallest on the least similar one (tau2-bench +3.6).
Capability: Using the tools it is given
Observed on
Qwen3 4B/8B/30B-A3B instruct backbones; environments grounded in a 15k-document code pretraining corpus; GRPO training for 400 rollouts; benchmarks held out from training but the g.
Sources
- Measured on three tool-use benchmarks (BFCL v4 multi-turn, tau2-bench, ACEBench-Agent) for three backbone sizes, each compared to its own untrained base. The base-model comparison is not compute-matched for the tool-use setting (the matched fixed-environment baselines were run only in the games setting), and the other synthetic-environment systems are transcribed from their own papers with different data, budget, and evaluation protocol. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For 1.5B-7B Qwen2.5 backbones trained with RL on multi-hop retrieval QA, giving the policy reversible memory operations (summary, backtrack over a stack) with per-action process rewards raises F1 over trajectory-reward agentic-RAG baselines, and removing the two memory actions costs more than removing the planning action.Carrying a chain of inference through to its conclusion · reviewed by AI
- On multi-turn tool-calling benchmarks that score whether the final environment state matches a gold trajectory, aggregate accuracy hides wrong action-class choices: a model can call a withheld or under-specified tool with fabricated arguments, be handed the missing piece on a later turn, and still be graded PASS — so heavily tool-trained model families rank high while almost never asking for a missing parameter or refusing an unavailable function.Whether the measurement made the finding · reviewed by AI
- For 4B-8B open base models mid-trained for tool use, grounding-oriented supervision (documentation/code turned into QA) and execution-oriented supervision (executable trajectories from real APIs and MCP skills) help different things, and only the combination improves everything: trajectories alone give the bigger function-calling gain while grounding alone transfers better to interactive and unseen-tool benchmarks, and each branch alone scores below no mid-training on MCP-Universe.Using the tools it is given · reviewed by AI
- When training 4B-8B Qwen3 agents with group-based RL (GRPO, DAPO, GSPO) on sparse-reward long-horizon tasks, injecting hints into the environment's feedback — procedural action guidance in early episode steps and early training, switching to enriched state observations in later steps and later training — raises success on the unmodified environments by a few points on average, while lowering performance on tasks that require questioning whether the request is answerable at all.Completing tasks that take many steps or many hours · reviewed by AI
- When fine-tuning small open models (4B-8B Qwen3) for multi-turn tool calling, supervising only the recovery steps generated after the first point where the student's rollout leaves the teacher-supported state region — and zeroing the loss on the student's own valid prefix — beats full-trajectory imitation and outcome-reward RL on in-domain and held-out tool benchmarks, and the trained student's successful trajectories get shorter across iterations rather than longer.Using the tools it is given · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Applying the same self-play environment-generation training and finding no improvement over a matched-compute fixed-environment baseline on multi-turn tool-calling benchmarks, or finding the gain uncorrelated with how closely a benchmark's task structure matches the generated environments. Proposed technique, not catalogued: Self-play RL over model-generated executable Gym-style environments.