region-guided-contrastive-decoding-medical-vlms-effectiveness-guidance-depends-how CLM-0473In region-guided contrastive decoding for medical VLMs, the effectiveness of the guidance depends on how much of the image the guiding mask covers: coarse ground-truth boxes (mean ~39% coverage on SLAKE) suppress so much of the unconditional branch that the contrast weakens, while compact automatically discovered masks (~7% coverage) selected by the drop in answer likelihood under occlusion give a sharper signal and higher closed-ended accuracy.
Capability: Grounding answers in the image
Observed on
Medical VQA (SLAKE, MIMIC-CXR) with three 7B-class Med-VLMs, inference-time contrastive decoding with classifier-free guidance on an anatomical region..
Sources
- Validated by: Overall/open/closed accuracy on SLAKE and MIMIC-CXR across Phi-3.5V-Med, HuatuoGPT-Vision-7B and LLaVA-Med-7B against plain decoding, VCD, DoLA, OPERA and GT-mask ARCD; ablations removing the area penalty (-3.4%), replacing counterfactual selection with random masks (-4.4%), and using a single occlusion mode.. Measured on two datasets and three Med-VLMs, with a coverage analysis on one model showing closed accuracy falling once GT coverage exceeds 40% and per-sample gains concentrated on large-GT cases. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In training-free contrastive decoding for 7B-class medical VLMs, the spatial and temporal contrast terms act on different answer types: anatomy-conditioned classifier-free guidance is what raises closed-ended accuracy (68.5 to 81.5 on SLAKE with Phi-3.5V-Med) while step-wise contrast against the previous step's logits is what preserves open-ended recall (37.8 to 44.9), so dropping either term costs about five overall points.Grounding answers in the image · reviewed by AI
- In 4-7B open vision-language models (LLaVA-1.5-7B, Qwen2.5-VL-7B, InternVL-2.5-4B), how the cross-modal attention distribution over visual tokens shifts between adjacent decoder layers separates hallucinated from grounded object predictions better than the model's sensitivity to masking the attended region, and better than attention drift measured against a fixed first or last layer anchor.Grounding answers in the image · reviewed by AI
- For binary visual question answering with open LVLMs (InstructBLIP-7B, Qwen2.5-VL, InternVL-4B), a hallucinated answer can be held with high response consistency, so a single consistency/uncertainty score misses it; separating consistency into four probe groups (image vs. text perturbation crossed with affirmative vs. negated statement) and classifying on the joint within- and between-group agreement pattern detects hallucination better than any single-score black-box or white-box baseline.Stating false facts confidently · reviewed by AI
- In LVLMs asked to describe people (LLaVA, InstructBLIP, Shikra), masking attention so that text tokens read visual tokens but language-only self-attention is suppressed in late decoder layers lowers gender misattribution while keeping answer coverage high, whereas visual-only or language-only masking scores well on the bias metric only because the model names a gender in under 10% of cases.Grounding answers in the image · reviewed by AI
- In frozen large vision-language models, token-level hallucination evidence is distributed across the relational, spatial and sequential structure of the model's internal trace rather than contained in any single token's hidden state, so a detector that reads the attention graph over image patches, query and response tokens with interleaved GNN/ViT/GRU modules localizes hallucinated spans better than flat probes on the same activations.Grounding answers in the image · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Matching the automatically selected masks' coverage to ground-truth boxes (or shrinking GT boxes to similar compactness) yields the same closed-ended accuracy, showing coverage is not what drives the gap; or random compact masks match counterfactually selected ones.