fine-tuning-small-open-models-4b-8b-qwen3-multi-turn-tool-calling CLM-0052When fine-tuning small open models (4B-8B Qwen3) for multi-turn tool calling, supervising only the recovery steps generated after the first point where the student's rollout leaves the teacher-supported state region — and zeroing the loss on the student's own valid prefix — beats full-trajectory imitation and outcome-reward RL on in-domain and held-out tool benchmarks, and the trained student's successful trajectories get shorter across iterations rather than longer.
Capability: Using the tools it is given
Observed on
4B and 8B Qwen3 students distilled from larger teachers on ~2,200 synthetic tool environments with verifiable feedback; evaluated on FTRL, BFCL multi-turn, ToolHop, tau-bench, RoTB.
Sources
- Averages over five benchmarks and two backbones against three distillation and four RL baselines, plus a stepwise component ablation showing self-distillation contributes the largest jump and localized masking, progressive iteration, and graph-based breakpoint detection add smaller increments; single lab, single training corpus, one teacher pool, no compute-matched baseline.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When fine-tuning a 14B model on knowledge-graph-derived multiple-choice QA for a single medical domain, attaching supporting triples from the same source text chunk to each target triple during supervision, and repairing residual one-hop failures before reinforcement learning, gives higher accuracy on held-out 3-, 4- and 5-hop questions than training on isolated triples or running RL from an unrepaired checkpoint.Carrying a chain of inference through to its conclusion · reviewed by AI
- For a small agent policy (Qwen3.5-4B) repeatedly deployed on the same task family with environment-level feedback, distilling only the trajectories whose textual-state guidance is newly or persistently useful relative to state-free execution raises the state-free policy's accuracy more than distilling all state-conditioned trajectories, and the resulting policy keeps improving over alternating rounds of state editing and consolidation.Improving its own scaffolding from failures · reviewed by AI
- Across 25 open-weight instruction-tuned models (4B-744B), the ability to modulate a named concept's residual-stream projection on plain instruction does not track parameter count or recency — the best scorer is an 8B 2024 model and some of the newest models score lowest — while within a single model line the score rises steadily across ordinary pre-training and post-training checkpoints, so activation controllability is acquired incidentally and must be measured per model rather than inferred from scale.Strategic deception and detecting it · reviewed by AI
- For 8B instruction-tuned backbones, training a LoRA corrector to diagnose a typed, executable error operator (error type, step location, corrupted and repaired proposition) and apply its inverse as a localized edit raises accuracy over free-form self-revision baselines (Self-Refine, RISE, SCoRe) on math, code and commonsense tasks, and dropping either the operator-prediction loss or the adjacent-state reconstruction loss loses most of the gain.Fixing its own mistakes · reviewed by AI
- For a frozen LLM run in a generate-critique-revise loop with no external feedback, the accuracy gain is concentrated where single-shot reasoning is already unreliable: on clean multi-step reasoning (BBH, ~74% single-shot) depth makes no difference beyond the confidence interval, while on harder math (MATH, 26.2% single-shot) the same loop gains +14.2 points, with GSM8K in between (+4.2).Fixing its own mistakes · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An ablation in which masking the loss to post-breakpoint steps performs no better than the same self-distillation data with full-trajectory loss, or where trajectory length grows with iterations while accuracy rises. Drafted stance toward tool-use-fine-tuning-closes-a-gap-rather-than-extending-a-frontier: supports -- The gains are again measured on small open models starting well below a larger teacher, with no evidence the procedure helps an already-strong model. Proposed technique, not catalogued: Loss-masked recovery-step self-distillation from the first divergence point.