medical-llms-given-clinical-practice-guideline-context-asked-produce CLM-0609observationsingle paperReviewed by AI
When medical LLMs are given a clinical practice guideline in context and asked to produce the decision pathway a case requires, final-outcome accuracy does not track adherence to the guideline-defined step sequence: models can reach the right outcome while omitting, reordering, skipping or padding steps, and medically fine-tuned models score no better on pathway consistency than the general models they were tuned from.
Capability: Competence in a specialist domain
Observed on
16 models on 7,458 cases built from 440 English and Chinese guidelines; guideline text supplied in context; pathway consistency measured by an LLM judge scoring step matches aggreg.
Sources
- Validated by: Pairwise human preference study on 200 sampled pathway pairs: the metric agreed with aggregated human preference 90.0% of the time, equal to the 90.0% agreement between two annotators; results held across three aggregation methods and two judge models (Kendall's W 0.70-0.87).. Measured across 16 models in single-turn and multi-turn settings. Qwen3-235B-A22B had the highest single-turn accuracy (75.81%) but not the lowest pathway cost (GPT-5.4 at 28.41%); Baichuan-M3-235B, fine-tuned from Qwen3-235B-A22B, had worse pathway cost in both settings. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In a deployed multilingual medical triage system, most of the accuracy gain comes from writing the clinicians' implicit decision tree into an explicit scoped rule set rather than from moving the final decision out of the LLM: replacing a flat danger-sign list with 231 structured rules in the same end-to-end prompt raised recall from 0.565 to 0.837 and F1 from 0.606 to 0.689, while further handing the decision to a deterministic rule engine over LLM-extracted symptoms left accuracy statistically unchanged (recall 0.810, F1 0.702).Following an unfamiliar procedure · reviewed by AI
- When injecting revised medical guideline facts into 4B-14B instruct models by LoRA SFT under a matched token budget, rendering the same update events as extended-matching items — several related vignettes sharing one candidate answer pool — transfers better to a held-out temporally anchored oncology staging benchmark than short-answer, fill-in-the-blank, or per-vignette multiple-select renderings of the identical content.Absorbing and flexibly using new knowledge · reviewed by AI
- For LLM agents on dependent multi-step tasks, success decays geometrically with the number of steps under a single per-step reliability parameter that rises with model scale but stays below 1, so long horizons eventually collapse even for the strongest deployed systems — and the decay tracks step count, not context length, since bounding the context window steepens the decay rather than easing it.Completing tasks that take many steps or many hours · reviewed by AI
- On complex clinical diagnosis from published case reports (rare disease and challenging cases), running several presenter-rebutter debater pairs on one backbone and synthesizing their revised hypotheses with an aggregator raises exact-match diagnostic accuracy and rated safety over the same model prompted once, while giving no measurable gain on multiple-choice clinical knowledge questions.Getting more from several models than from one · reviewed by AI
- Having a model score its confidence step by step detects failures in multi-step reasoning better than one holistic confidence score for the whole answer, with up to 15% higher AUROC on two multi-step benchmarks, so where a chain fails is more visible per step than in the final answer.Carrying a chain of inference through to its conclusion · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Showing that outcome accuracy and pathway consistency rank models the same way on this benchmark, or that medically fine-tuned models consistently beat their base models on pathway consistency.