7-13b-open-vision-language-models-object-hallucination-explained-weak-visual CLM-0532In 7-13B open vision-language models, object hallucination is not explained by weak visual attention e hallucinated and real object tokens draw comparable attention to specific image regions in mid-to-late layers; the separating signal is whether a logit-lens decode of those attended visual tokens matches the generated object word.
Capability: Grounding answers in the image
Observed on
Open LVLMs with accessible attention and hidden states (LLaVA-1.5 7B/13B, Shikra-7B, Qwen2-VL-7B); free-form image captioning on COCO/AMBER; object-level hallucination only..
Sources
- Validated by: Detection on 500 COCO images with CHAIR labels: F1 0.7932 vs 0.6842 (SVAR), 0.6811 (InterConf), 0.6182 (Uncertainty Score). Mitigation: lowest CHAIR_S/CHAIR_I on four LVLMs (e.g. 26.8/10.0 vs PAI 29.8/13.2 on LLaVA-1.5-7B) and lowest AMBER CHAIR/Hal while keeping Cover at or above the greedy baseline.. Measured on 500 COCO images with LLaVA-1.5-7B for the attention and logit-lens analysis, with masking interventions splitting hallucinations roughly 2:1 into visual-uncertainty vs contextual-prior. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For short-answer object-existence questions in open multimodal models (LLaVA v1.5, MiniGPT-4, Qwen2.5-VL), object hallucination is driven partly by the visual encoder itself rather than only by language priors: hallucinated cases show lower image-text embedding similarity and inverted attention, and swapping in a weaker or noisier visual encoder lowers POPE accuracy while a stronger one raises it.Grounding answers in the image · reviewed by AI
- In 7-8B multimodal models, object hallucination during visually grounded token generation tracks a shift of the visual-language information ratio toward language inside mixed-modality "synergy" attention heads rather than a shortage of visual-specific heads, and rescaling visual versus language value vectors in those heads causally moves hallucination in both directions.Grounding answers in the image · reviewed by AI
- For short-answer object-existence questions in open multimodal models (LLaVA v1.5, MiniGPT-4, Qwen2.5-VL), object hallucination is driven partly by the visual encoder itself rather than only by language priors: hallucinated cases show lower image-text embedding similarity and inverted attention, and swapping in a weaker or noisier visual encoder lowers POPE accuracy while a stronger one raises it.Grounding answers in the image · reviewed by AI
- In 7B open vision-language models (LLaVA-1.5, Shikra), the per-token log-probability gap between a normal forward pass and one with visual features zeroed shrinks as a caption gets longer, and while that gap ranks hallucinated tokens slightly better than entropy or max-probability, its absolute separation is weak (AUROC around 0.57-0.59) and it fails entirely when aggregated to sentence level.Grounding answers in the image · reviewed by AI
- In 7B vision-language models that prune visual tokens during decoding for efficiency, hallucination rises as the generated caption gets longer, because pruning is irreversible and discards fine-grained evidence needed at later decoding steps; keeping the pruned tokens as a retrievable bank and pulling them back when next-token entropy spikes recovers the grounding without a second forward pass.Grounding answers in the image · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Measuring image-attention magnitude at object tokens in these models and finding hallucinated tokens systematically attend less than real ones, or finding logit-lens decodes of attended patches match hallucinated words as often as real ones. Drafted stance toward 7-8b-multimodal-models-object-hallucination-during-visually-grounded-token: contests -- That claim ties hallucination to a shift of the visual-language information ratio inside attention heads, whereas this paper measures no systematic difference in visual attention magnitude between real and hallucinated object tokens and locates the failure in the semantics of the attended region instead.