injecting-revised-medical-guideline-facts-into-4b-14b-instruct-models CLM-0579When injecting revised medical guideline facts into 4B-14B instruct models by LoRA SFT under a matched token budget, rendering the same update events as extended-matching items — several related vignettes sharing one candidate answer pool — transfers better to a held-out temporally anchored oncology staging benchmark than short-answer, fill-in-the-blank, or per-vignette multiple-select renderings of the identical content.
Capability: Absorbing and flexibly using new knowledge
Observed on
Oncology guideline updates, LoRA SFT, Qwen3 1.7B-14B plus Llama-3.1-8B; formats still differ in candidate-set size and surface complexity despite matched token budget; not uniforml.
Sources
- Validated by: SEER-Bench answer accuracy 57.6 to 64.8 and rationale accuracy 50.7 to 59.6 on Qwen3-4B, versus 61.5-62.7 answer accuracy for the three other same-budget formats; HealthBench retention 0.247 to 0.263; replicated on Llama-3.1-8B and across 1.7B/8B/14B backbones; vignette-matched and density-matched controls reported in appendices.. Measured on a single oncology benchmark built by the same authors (1,992 cases, expert-reviewed) plus HealthBench Professional and a MedGUIDE diagnostic. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On complex clinical diagnosis from published case reports (rare disease and challenging cases), running several presenter-rebutter debater pairs on one backbone and synthesizing their revised hypotheses with an aggregator raises exact-match diagnostic accuracy and rated safety over the same model prompted once, while giving no measurable gain on multiple-choice clinical knowledge questions.Getting more from several models than from one · reviewed by AI
- A medically fine-tuned Llama-3.1-8B checkpoint (UltraMedical) scores below its general-purpose Instruct counterpart on both selecting the correct definition of medical jargon in EHR context and judging whether a term is jargon, and its errors are almost entirely false positives — it treats ordinary words in clinical text as specialized.Competence in a specialist domain · reviewed by AI
- For detecting findings-conclusion inconsistencies in oncology PET/CT radiology reports, a 15M-parameter BERT pre-trained on in-domain reports reaches higher balanced accuracy and lower false-positive rate than zero- or few-shot prompted 27B-70B open-weight LLMs, and LoRA fine-tuning the 70B model only matches the small model rather than beating it.Spotting an error in a given artifact · reviewed by AI
- When medical LLMs are given a clinical practice guideline in context and asked to produce the decision pathway a case requires, final-outcome accuracy does not track adherence to the guideline-defined step sequence: models can reach the right outcome while omitting, reordering, skipping or padding steps, and medically fine-tuned models score no better on pathway consistency than the general models they were tuned from.Competence in a specialist domain · reviewed by AI
- When injecting new or counterfactual facts into a 3-4B instruct model by post-training, supervised fine-tuning on extracted QA pairs binds facts to their training question form — recall of the single fact stays reasonable while multi-source and inferential questions over the same facts collapse — and adding more SFT epochs does not close the gap, whereas mixed-policy RL that supplies the golden answer as an off-policy trajectory when on-policy rollouts fail recovers much of the generalization.Absorbing and flexibly using new knowledge · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A matched-budget comparison on a different domain or update algorithm where extended-matching supervision transfers no better than short-answer or fill-in-the-blank rendering of the same facts, or where the gain disappears once candidate-set size and surface complexity are equalized.