complex-clinical-diagnosis-published-case-reports-rare-disease-challenging CLM-0555On complex clinical diagnosis from published case reports (rare disease and challenging cases), running several presenter-rebutter debater pairs on one backbone and synthesizing their revised hypotheses with an aggregator raises exact-match diagnostic accuracy and rated safety over the same model prompted once, while giving no measurable gain on multiple-choice clinical knowledge questions.
Capability: Getting more from several models than from one
Observed on
GPT-4o, GPT-3.5 and GPT-4o-mini backbones; 297 rare disease and 1,719 challenging published cases; accuracy graded by exact match to the reference diagnosis and safety by an LLM ju.
Sources
- supportsA Structured Debate-Mixture-of-Agents Framework for Complex Clinical Diagnostic Decision SupportValidated by: Top-1/top-3 exact-match accuracy and rubric safety rate against single-model prompting, a single model prompted to imitate the debate, MoA with the rebutter removed, and a MAC-style single-debater multi-round dialogue; on challenging cases GPT-4o rose from 14.50% to 25.34% top-1 accuracy and 63.46% to 76.14% top-1 safety, with McNemar p<0.01, and all three ablations scored below the full topology.. Measured on two case sets with paired one-sided McNemar tests and three ablation variants across three backbones; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Asked to draft clinical-guideline content against expert references, 17 models and deep-research agents reach at best about 57% accuracy on clinical claims, average under 30% on mechanistic claims and about 22% on differential diagnosis, and stay below 30% on therapeutic claims even for the best system, so expert-level evidence integration in medicine is well beyond current systems.Competence in a specialist domain · reviewed by AI
- When injecting revised medical guideline facts into 4B-14B instruct models by LoRA SFT under a matched token budget, rendering the same update events as extended-matching items — several related vignettes sharing one candidate answer pool — transfers better to a held-out temporally anchored oncology staging benchmark than short-answer, fill-in-the-blank, or per-vignette multiple-select renderings of the identical content.Absorbing and flexibly using new knowledge · reviewed by AI
- A medically fine-tuned Llama-3.1-8B checkpoint (UltraMedical) scores below its general-purpose Instruct counterpart on both selecting the correct definition of medical jargon in EHR context and judging whether a term is jargon, and its errors are almost entirely false positives — it treats ordinary words in clinical text as specialized.Competence in a specialist domain · reviewed by AI
- For LLM agents doing active clinical diagnosis in interactive patient environments, wrapping the model in a scaffold of role-specialized agents, evidence tools and an explicit evidence state, with a runtime observer that tracks uncertainty and evidence coverage to decide when to stop acquiring evidence, raises both diagnostic accuracy and execution stability over static case-to-answer prompting, with the size of the gain depending on the backbone model.Using the tools it is given · reviewed by AI
- When medical LLMs are given a clinical practice guideline in context and asked to produce the decision pathway a case requires, final-outcome accuracy does not track adherence to the guideline-defined step sequence: models can reach the right outcome while omitting, reordering, skipping or padding steps, and medically fine-tuned models score no better on pathway consistency than the general models they were tuned from.Competence in a specialist domain · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A matched-compute single-model baseline (same token spend, repeated sampling and self-critique) reaching the same accuracy and safety on these case sets, or the gain vanishing when the exact-match grader is replaced with a less brittle one.