1-5b-8b-base-models-given-domain-sft-expert-math CLM-0339For 1.5B-8B base models given a domain SFT expert (math or medical), keeping both models and routing token by token — invoking the expert only on tokens where a trained router predicts the base would fail — preserves general-benchmark accuracy within about 0.5% of the base while exceeding the expert's in-domain accuracy, whereas loss-modifying SFT regularizers still lose up to 16% general accuracy.
Capability: Retaining skills and safety after fine-tuning
Observed on
Gemma3-4B, Llama3.1-8B, Qwen2.5-1.5B; math (GSM8K-trained) and medical (PubMedQA-trained) experts; MMLU/CSQA/ARC-C as general tests; requires holding both models in memory and an e.
Sources
- Validated by: Four model-domain settings, six benchmarks; compared against vanilla SFT, DFT, EAFT, LfU, RouteLLM, Switch Generation, ensemble and contrastive decoding sharing the same base and expert; highest overall average in all settings, plus ablations on router architecture, momentum and thresholds, and wall-clock latency (1.40-1.44x expert-only vs 1.89-2.25x ensemble).. Measured on two backbones by two domains, six benchmarks each, with matched base/expert across all baselines; external-expert generalization tested on two public experts. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When 7-8B open-weight instruct models are fully fine-tuned on a single domain dataset, confining gradient updates to FFN neurons that activate stably across the training set and adding an MSE loss that pulls activations back toward pre-computed activation anchors retains far more general-benchmark ability (MMLU, BBH, TyDiQA) than full-parameter fine-tuning, by about 9.9 points on average, while also raising in-domain scores.Retaining skills and safety after fine-tuning · reviewed by AI
- For 4B-8B open base models mid-trained for tool use, grounding-oriented supervision (documentation/code turned into QA) and execution-oriented supervision (executable trajectories from real APIs and MCP skills) help different things, and only the combination improves everything: trajectories alone give the bigger function-calling gain while grounding alone transfers better to interactive and unseen-tool benchmarks, and each branch alone scores below no mid-training on MCP-Universe.Using the tools it is given · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- For a same-family pair of reasoning and instruction-tuned 4B models, learning one interpolation ratio per module (attention block, FFN block, embedding, final norm) with a reward that scores correctness and ranks correct responses by length cuts average generated tokens by about 27% relative to the reasoning model while holding average accuracy, and beats fixed global-ratio merging and CMA-ES search over the same ratios.Cutting to the chase · reviewed by AI
- On a suite of verifier-checked tool-using ReAct task families (arithmetic, SQL analytics, security judgment, URL grounding, dependency planning, orchestration, tool policy), a single task-weighted aggregate accuracy hides inverted per-family rankings: the model that ranks last overall can lead both URL-grounding families while scoring lowest on structured SQL and multi-step tool families, which spread the four tested models by more than sixty points.Whether the measurement made the finding · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A setting where token-level routing between base and expert either fails to recover general-capability loss (general accuracy still well below base) or fails to match the expert in domain accuracy, while a loss-side regularizer does both.