medically-fine-tuned-llama-3-1-8b-checkpoint-ultramedical-scores-below-general-purpose CLM-0439A medically fine-tuned Llama-3.1-8B checkpoint (UltraMedical) scores below its general-purpose Instruct counterpart on both selecting the correct definition of medical jargon in EHR context and judging whether a term is jargon, and its errors are almost entirely false positives — it treats ordinary words in clinical text as specialized.
Capability: Competence in a specialist domain
Observed on
One model pair (Llama-3.1-8B Instruct vs UltraMedical), English medical jargon benchmarks built from README/UMLS; fine-tuning data and procedure are uncontrolled confounds..
Sources
- Validated by: Raised UltraMedical's jargon-identification accuracy from 53.1% to 56.1% (McNemar p=0.01) by correcting 270 false positives at the cost of 146 new false negatives; the same procedure lowered Instruct's accuracy by 1.0 point and gave under 0.2 points on the understanding task.. Measured on held-out splits (1,495 JU items, 2,990 JI items) with Wilson intervals and a paired McNemar test; gaps are small (76.1 vs 74.1 on JU, 58.8 vs 53.1 on JI) and the design cannot isolate fine-tuning as the cause, since training data and recipe differ.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On complex clinical diagnosis from published case reports (rare disease and challenging cases), running several presenter-rebutter debater pairs on one backbone and synthesizing their revised hypotheses with an aggregator raises exact-match diagnostic accuracy and rated safety over the same model prompted once, while giving no measurable gain on multiple-choice clinical knowledge questions.Getting more from several models than from one · reviewed by AI
- When injecting revised medical guideline facts into 4B-14B instruct models by LoRA SFT under a matched token budget, rendering the same update events as extended-matching items — several related vignettes sharing one candidate answer pool — transfers better to a held-out temporally anchored oncology staging benchmark than short-answer, fill-in-the-blank, or per-vignette multiple-select renderings of the identical content.Absorbing and flexibly using new knowledge · reviewed by AI
- For detecting findings-conclusion inconsistencies in oncology PET/CT radiology reports, a 15M-parameter BERT pre-trained on in-domain reports reaches higher balanced accuracy and lower false-positive rate than zero- or few-shot prompted 27B-70B open-weight LLMs, and LoRA fine-tuning the 70B model only matches the small model rather than beating it.Spotting an error in a given artifact · reviewed by AI
- When an LLM judge is given a prior evaluation score as context metadata on a revision, its rubric scores shift toward that uninformative prior — seven of eight tested models (API and open-weight, 0-5 scale, 20 fixed texts) show a negative total shift, and on a categorical compliance task the metadata blocks 48% of error corrections and flips about 10% of previously correct judgments to the anchored wrong label.Judging another's work as well as a careful reviewer would · reviewed by AI
- When medical LLMs are given a clinical practice guideline in context and asked to produce the decision pathway a case requires, final-outcome accuracy does not track adherence to the guideline-defined step sequence: models can reach the right outcome while omitting, reordering, skipping or padding steps, and medically fine-tuned models score no better on pathway consistency than the general models they were tuned from.Competence in a specialist domain · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A controlled fine-tune on medical text, or other general/medical model pairs, showing the domain-adapted model matching or beating the general model on the same jargon understanding and identification tasks.