llm-agents-doing-active-clinical-diagnosis-interactive-patient-environments CLM-0585For LLM agents doing active clinical diagnosis in interactive patient environments, wrapping the model in a scaffold of role-specialized agents, evidence tools and an explicit evidence state, with a runtime observer that tracks uncertainty and evidence coverage to decide when to stop acquiring evidence, raises both diagnostic accuracy and execution stability over static case-to-answer prompting, with the size of the gain depending on the backbone model.
Capability: Using the tools it is given
Observed on
Interactive diagnostic environments synthesized from raw clinical cases; evaluated on execution robustness, reasoning dynamics and diagnostic outcomes..
Sources
- Validated by: A three-level evaluation pyramid (execution robustness, reasoning dynamics, diagnostic outcomes) reporting improved diagnostic performance and process stability over baselines; the abstract gives no numbers.. Abstract only: claims improvement in diagnostic performance and process stability with model-dependent boundaries, but reports no metrics, baselines, case counts or ablations, so the contribution of the stopping harness versus the role scaffold is not isolated.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On complex clinical diagnosis from published case reports (rare disease and challenging cases), running several presenter-rebutter debater pairs on one backbone and synthesizing their revised hypotheses with an aggregator raises exact-match diagnostic accuracy and rated safety over the same model prompted once, while giving no measurable gain on multiple-choice clinical knowledge questions.Getting more from several models than from one · reviewed by AI
- For a frozen-backbone multi-agent scientific discovery pipeline on HypoBench tasks, distilling reusable procedural skills separately per stage — from discriminator critique of a whole hypothesis batch for generation, and from execution outcomes compared to ground-truth results for testing — beats no skill, an LLM-written skill, and a hand-written skill, and removing each stage's feedback signal costs most of the gain.Improving its own scaffolding from failures · reviewed by AI
- Asked to draft clinical-guideline content against expert references, 17 models and deep-research agents reach at best about 57% accuracy on clinical claims, average under 30% on mechanistic claims and about 22% on differential diagnosis, and stay below 30% on therapeutic claims even for the best system, so expert-level evidence integration in medicine is well beyond current systems.Competence in a specialist domain · reviewed by AI
- For single-agent procedural tasks where the relevant state fits a fixed schema known in advance, replacing the appended conversation transcript with a structured execution state that the model rewrites each step — discarding the reasoning trace once the state patch is validated — holds accuracy flat as the horizon grows while history-appending runtimes degrade, and the gain comes from the structure rather than the shorter prompt.Completing tasks that take many steps or many hours · reviewed by AI
- When automatically rewriting the prompts of a multi-agent LLM system from its failures, locating the responsible agent by injecting a hint into one agent at a time and using that agent's hint-corrected intermediate output as the supervision signal produces prompt updates that generalize better than critique derived only from the system-level output.Improving its own scaffolding from failures · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Running the same backbones on the same synthesized environments with and without the scaffold and harness and finding no accuracy or stability gain, or finding the gain comes from the environment construction alone rather than the role agents and stopping rule.