asked-draft-clinical-guideline-content-against-expert-references-17-models CLM-0291observationsingle paperReviewed by AI
Asked to draft clinical-guideline content against expert references, 17 models and deep-research agents reach at best about 57% accuracy on clinical claims, average under 30% on mechanistic claims and about 22% on differential diagnosis, and stay below 30% on therapeutic claims even for the best system, so expert-level evidence integration in medicine is well beyond current systems.
Capability: Competence in a specialist domain
Observed on
MedProbeBench, guideline-grounded tasks with 1,200+ rubric criteria and 5,130+ atomic claims, 2026; 17 LLMs and deep-research agents; claim-level verification..
Sources
- Rubric and claim verification are themselves partly model-run; the benchmark is new and single-authored-group. The category ordering (clinical > mechanistic > therapeutic) is the informative part.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On complex clinical diagnosis from published case reports (rare disease and challenging cases), running several presenter-rebutter debater pairs on one backbone and synthesizing their revised hypotheses with an aggregator raises exact-match diagnostic accuracy and rated safety over the same model prompted once, while giving no measurable gain on multiple-choice clinical knowledge questions.Getting more from several models than from one · reviewed by AI
- When asked to supply a missing biomedical reference, models that retrieve a real paper often still misstate its journal, year or authors, so verifiable-reference rate overstates usable accuracy: two Claude models with near-identical verifiable rates (77.6% and 76.6%) were correct in every bibliographic field in 54.6% and 19.9% of responses.Stating false facts confidently · reviewed by AI
- For LLM agents doing active clinical diagnosis in interactive patient environments, wrapping the model in a scaffold of role-specialized agents, evidence tools and an explicit evidence state, with a runtime observer that tracks uncertainty and evidence coverage to decide when to stop acquiring evidence, raises both diagnostic accuracy and execution stability over static case-to-answer prompting, with the size of the gain depending on the backbone model.Using the tools it is given · reviewed by AI
- In LLM fact-checking over supplied evidence, high veracity accuracy does not imply the verdict depends on that evidence: when the model's own cited sentences are iteratively removed, off-the-shelf models (GPT-4o-mini, Gemini-2.5-Flash-Lite, Qwen-2.5-32B, Llama-3.1-8B) keep their SUPPORT/REFUTE labels at a non-zero plateau instead of abstaining, and training with paired supervision that requires NOT_ENOUGH_INFO on the evidence-ablated input removes most of that persistence without losing accuracy.Checking claims against evidence · reviewed by AI
- When medical LLMs are given a clinical practice guideline in context and asked to produce the decision pathway a case requires, final-outcome accuracy does not track adherence to the guideline-defined step sequence: models can reach the right outcome while omitting, reordering, skipping or padding steps, and medically fine-tuned models score no better on pathway consistency than the general models they were tuned from.Competence in a specialist domain · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A system exceeding 70% claim accuracy across mechanistic, diagnostic and therapeutic categories on this benchmark under the same verification.