arxiv-2608-19197 SRC-0099 · paper

SPADE: Self-Play in Adaptive Synthetic Executable Environments

Created: 2026-08-19 · Ingested: 2026-09-15

https://arxiv.org/abs/2608.19197

Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that lear

In brief

A model can generate its own training environments as executable code and improve more than training on a fixed pool, if the generator is itself trained online. SPADE has one LLM alternate between an Environment Designer, which writes complete Gym-style Python MDPs plus a privileged hint, and a Reasoning Agent that plays them; the Designer is rewarded by the return gap between hinted and unhinted plays, a proxy for minimax regret.

Trained on Qwen3-4B, 8B and 30B-A3B with GRPO for 400 rollouts of 24 environments, evaluated on 8 held-out benchmarks (AIME 2025/2026, GPQA-Diamond, LiveCodeBench-v6, 4 Reasoning-Gym categories) and 3 tool-use suites.

At 30B-A3B the suite average rises to 58.3, +8.1 over base and +5.3 over the strongest fixed-environment baseline; gains concentrate in procedural reasoning (RG-Math +18.3) while competition math is roughly flat. Tool use gains +13.9 on ACEBench-Agent and +5.7 on BFCL v4 multi-turn. Gains grow with scale: +5.2 at 4B, +5.7 at 8B.

Baselines are retrained per backbone at the same 400-iteration budget, which is the strongest part of the design. Ablations isolate the reward (regret +8.1 vs EMA learning potential +5.7) and corpus grounding (Vendi/n 0.68 vs 0.04), but the no-Designer-training control also removes memory, so those two are confounded. The hint-regret estimate goes negative for long stretches at 4B and 8B. Tool-use comparisons to AgentScaler, EnvScaler and others are transcribed from papers with different data, budgets and protocols, so they are not controlled. 1 model family, no larger dense models.

If you assumed synthetic environment pools were the ceiling, this argues the generator should be trained too.

Written from the abstract by claude-opus-5 on 2026-09-15, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsFor 4B-30B Qwen3 backbones trained with self-play RL where the same model writes executable multi-turn tool environments (simulated tool schemas, backend state, per-instruction state checks) and then solves them, multi-turn function-calling scores rise, and the gain is largest on the benchmark whose task structure most closely mirrors the generated environments (ACEBench-Agent +13.9 at 30B-A3B) and smallest on the least similar one (tau2-bench +3.6).