detecting-findings-conclusion-inconsistencies-oncology-pet-ct-radiology-reports-15m-parameter CLM-0559For detecting findings-conclusion inconsistencies in oncology PET/CT radiology reports, a 15M-parameter BERT pre-trained on in-domain reports reaches higher balanced accuracy and lower false-positive rate than zero- or few-shot prompted 27B-70B open-weight LLMs, and LoRA fine-tuning the 70B model only matches the small model rather than beating it.
Capability: Spotting an error in a given artifact
Observed on
Synthetic errors (laterality, site, severity, disease status, whole-conclusion swap) inserted into the conclusion of 3,000 held-out real PET/CT reports; domain model pre-trained on.
Sources
- Validated by: Balanced accuracy on an 11,500-report synthetic benchmark: BERT-6 94.4% / FPR 5.8% vs 84.0% for the best prompted LLM (Llama-3.3-70B, prompt P7); LoRA-tuned Llama-3.3-70B 94.4%. Ablation shows MLM pre-training supplies most of the gain (BERT-6 56.5% to 93.7%). Lexical null models reach only 69.9% balanced accuracy, ruling out keyword artefacts.. Measured on 11,500 held-out reports with synthetic errors, four BERT sizes against three LLMs under 12 prompt variants, plus pre-training ablation and lexical null baselines. Errors are synthetic and conclusion-only; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When injecting revised medical guideline facts into 4B-14B instruct models by LoRA SFT under a matched token budget, rendering the same update events as extended-matching items — several related vignettes sharing one candidate answer pool — transfers better to a held-out temporally anchored oncology staging benchmark than short-answer, fill-in-the-blank, or per-vignette multiple-select renderings of the identical content.Absorbing and flexibly using new knowledge · reviewed by AI
- When an LLM judge is given a prior evaluation score as context metadata on a revision, its rubric scores shift toward that uninformative prior — seven of eight tested models (API and open-weight, 0-5 scale, 20 fixed texts) show a negative total shift, and on a categorical compliance task the metadata blocks 48% of error corrections and flips about 10% of previously correct judgments to the anchored wrong label.Judging another's work as well as a careful reviewer would · reviewed by AI
- On complex clinical diagnosis from published case reports (rare disease and challenging cases), running several presenter-rebutter debater pairs on one backbone and synthesizing their revised hypotheses with an aggregator raises exact-match diagnostic accuracy and rated safety over the same model prompted once, while giving no measurable gain on multiple-choice clinical knowledge questions.Getting more from several models than from one · reviewed by AI
- In a deployed multilingual medical triage system, most of the accuracy gain comes from writing the clinicians' implicit decision tree into an explicit scoped rule set rather than from moving the final decision out of the LLM: replacing a flat danger-sign list with 231 structured rules in the same end-to-end prompt raised recall from 0.565 to 0.837 and F1 from 0.606 to 0.689, while further handing the decision to a deterministic rule engine over LLM-extracted symptoms left accuracy statistically unchanged (recall 0.810, F1 0.702).Following an unfamiliar procedure · reviewed by AI
- When medical LLMs are given a clinical practice guideline in context and asked to produce the decision pathway a case requires, final-outcome accuracy does not track adherence to the guideline-defined step sequence: models can reach the right outcome while omitting, reordering, skipping or padding steps, and medically fine-tuned models score no better on pathway consistency than the general models they were tuned from.Competence in a specialist domain · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A prompted or lightly adapted frontier LLM matching the compact in-domain model's balanced accuracy and false-positive rate on this or a comparable PET/CT error benchmark, or the compact model failing to transfer to naturally occurring errors or another hospital's reports.