arxiv-2604-18418 SRC-0350 · paper

MedProbeBench: Systematic Benchmarking at Deep Evidence Integration for Expert-level Medical Guideline

Created: 2026-04-20 · Ingested: 2026-09-19

https://arxiv.org/abs/2604.18418(opens in a new tab)

Recent advances in deep research systems enable large language models to retrieve, synthesize, and reason over large-scale external knowledge. In medicine, developing clinical guidelines critically depends on such deep evidence integration. However, existing benchmarks fail to evaluate this capability in realistic workflows requiring multi-step evidence integration and expert-level judgment. To address this gap, we introduce MedProbeBench, the first benchmark leveraging high-quality clinical guidelines as expert-level references. Medical guidelines, with their rigorous standards in neutrality

In brief

Deep research agents write clinical guidelines that read well and cite consistently while failing to recover the evidence that substantiates their claims. The gap is largest exactly where clinical value lies: mechanism, differential diagnosis, and treatment.

MedProbeBench builds 50 guideline-generation tasks from 5 volumes of the WHO Classification of Tumours (5th Edition), each requiring a full multi-section document from a disease name and schema. Grading uses 1,236 task-adaptive rubric criteria scored by GPT-4.1 plus claim-level checks against 5,133 atomic claims, of which 2,203 (42.92%) carry citations. 17 systems were run, split between LLMs with search tools and deep research agents.

The best system, MiroThinker-v1.5-pro, scores 0.631 overall: 0.772 holistic quality but 0.570 on evidence verification, with task success rate 0.452 and search effectiveness 0.496. Averaged across systems, mechanistic claims hit 28.8%, differential diagnosis 22.2%, therapeutic claims 19.2%. Tongyi DR-30B (0.348) trails most LLM+search baselines, so architecture class alone does not predict rank. Thinking variants of Claude-Sonnet 4 and Gemini-3-Flash score below their standard versions.

Evidenced by measurement, with human validation: 40 samples scored by 3 senior experts (Spearman rho = 0.754, p = 0.012) and 200 claim-reference pairs (F1 86.4%, Cohen's kappa = 0.79). Each system was run once per task, so no variance estimates, and the judge is the same model family as one evaluated system.

Scope is oncology guidelines only. Treat high citation consistency as no evidence of grounding.

Written from the abstract by claude-opus-5 on 2026-10-17, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.