detecting-ungrounded-object-mentions-open-ended-captions-white-box-vision-language-models CLM-0042For detecting ungrounded object mentions in open-ended captions from white-box vision-language models, a single internal support signal (token likelihood, attention share, or image-text similarity) confuses support that comes from object-specific image evidence with support that comes from scene plausibility or the generated text prefix; scoring both whether image tokens carry object-specific evidence and whether support is more image-derived than prefix-derived separates these and raises detection AUROC.
Capability: Grounding answers in the image
Observed on
Object-existence hallucinations only, COCO/VOC category vocabularies, greedy-decoded detailed-description prompts, and access to internal hidden states; not tested on attributes, r.
Sources
- Object-level AUROC/AUPR on MSCOCO and Pascal VOC over eight LVLMs (LLaVA-1.5-7B/13B, LLaVA-NeXT, InstructBLIP, MiniGPT-4, InternVL3, Shikra, Qwen2.5-VL) against six training-free baselines; best average AUROC on both datasets. Ablations show compatibility alone at 72.25 AUROC rising to 83.88 with the evidence gate on LLaVA-1.5-7B, and counterfactuals (blank image, patch shuffle, prefix removal) move each component in the predicted direction. Single group, one prompt, one decoding setting; 500 images per benchmark.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On long-video (10-150 minute) audio-visual reasoning questions, multiple-choice accuracy overstates grounding: a large open model can answer about 55% of items correctly while scoring 8.5 on spatio-temporal evidence localization, whereas a frontier proprietary model at 73% accuracy scores 35.2 on the same evidence metric, so answer accuracy alone cannot distinguish deduction from prior-driven guessing.Reasoning about time in video · reviewed by AI
- When Video-LLMs are asked to verify sentence by sentence whether a dense video caption is supported by the video, they detect hallucinated sentences less often in captions they generated themselves than in captions from other models, and detection of temporal errors (action, order, causality, frequency, state change) improves much less with model scale than detection of static-frame errors.Reasoning about time in video · reviewed by AI
- In frozen large vision-language models, token-level hallucination evidence is distributed across the relational, spatial and sequential structure of the model's internal trace rather than contained in any single token's hidden state, so a detector that reads the attention graph over image patches, query and response tokens with interleaved GNN/ViT/GRU modules localizes hallucinated spans better than flat probes on the same activations.Grounding answers in the image · reviewed by AI
- In 4-7B open vision-language models (LLaVA-1.5-7B, Qwen2.5-VL-7B, InternVL-2.5-4B), how the cross-modal attention distribution over visual tokens shifts between adjacent decoder layers separates hallucinated from grounded object predictions better than the model's sensitivity to masking the attended region, and better than attention drift measured against a fixed first or last layer anchor.Grounding answers in the image · reviewed by AI
- For deepfake video detection with 7-8B open multimodal backbones, splitting the analysis into four separately prompted perspective agents (texture, lighting, motion, physics) whose reports a judge agent reconciles beats the same backbone doing chain-of-thought or multi-turn self-questioning over the same four perspectives, and the gain is not explained by extra turns or repeated frame access.Getting more from several models than from one · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A detector using a single internal support signal matching or beating the two-sided score across the same models and datasets, or ablations showing the evidence gate and the image-vs-prefix ratio add nothing over compatibility alone. Proposed technique, not catalogued: two-sided visual evidence and reliance scoring.