arxiv-2608-17427 SRC-0420 · paperCounterfactual Anatomy-guided Spatial-Temporal Decoding for Annotation-Free Hallucination Mitigation in Medical VLMs
Yifan Lu, Adinath Dukre, Abhijit Das, Ziyun Zou, Haolin Yang, Yutong Xie
Created: 2026-08-18 · Ingested: 2026-10-17
https://arxiv.org/abs/2608.17427(opens in a new tab)Medical vision-language models (Med-VLMs) have demonstrated strong performance on medical visual question answering, yet they remain prone to hallucination, generating clinically unsupported statements that are insufficiently grounded in image evidence. Mitigation methods applied during decoding offer a practical solution, but they typically lack anatomical awareness or rely heavily on ground truth annotations, which limits their applicability. We propose Counterfactual Anatomy-guided Spatial-Temporal decoding (CAST), a framework that operates entirely during inference and requires no manual a
Referenced by
Claims in this catalog that draw on this source, and whether as support or counterpoint.
- supportsIn region-guided contrastive decoding for medical VLMs, the effectiveness of the guidance depends on how much of the image the guiding mask covers: coarse ground-truth boxes (mean ~39% coverage on SLAKE) suppress so much of the unconditional branch that the contrast weakens, while compact automatically discovered masks (~7% coverage) selected by the drop in answer likelihood under occlusion give a sharper signal and higher closed-ended accuracy.
- supportsIn training-free contrastive decoding for 7B-class medical VLMs, the spatial and temporal contrast terms act on different answer types: anatomy-conditioned classifier-free guidance is what raises closed-ended accuracy (68.5 to 81.5 on SLAKE with Phi-3.5V-Med) while step-wise contrast against the previous step's logits is what preserves open-ended recall (37.8 to 44.9), so dropping either term costs about five overall points.