short-answer-object-existence-questions-open-multimodal-models-llava-v15-minigpt-4 CLM-0150For short-answer object-existence questions in open multimodal models (LLaVA v1.5, MiniGPT-4, Qwen2.5-VL), object hallucination is driven partly by the visual encoder itself rather than only by language priors: hallucinated cases show lower image-text embedding similarity and inverted attention, and swapping in a weaker or noisier visual encoder lowers POPE accuracy while a stronger one raises it.
Capability: Grounding answers in the image
Observed on
Binary yes/no existence questions with very short outputs (POPE, MME-Existence); the paper says long-form description settings are more affected by language priors..
Sources
- supportsBeyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLMCosine-similarity comparison and Grad-CAM entropy over 500 hallucinated vs 500 correct samples on LLaVA v1.5, plus encoder-swap interventions on POPE (average accuracy 0.842 baseline, 0.739-0.822 degraded, 0.864 with SigLIP-SO400M). Entropy differences are small (5.1%, 6.2%) and the diagnostic analysis is on one model; the encoder interventions do isolate visual feature quality.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For short-answer object-existence questions in open multimodal models (LLaVA v1.5, MiniGPT-4, Qwen2.5-VL), object hallucination is driven partly by the visual encoder itself rather than only by language priors: hallucinated cases show lower image-text embedding similarity and inverted attention, and swapping in a weaker or noisier visual encoder lowers POPE accuracy while a stronger one raises it.Grounding answers in the image · reviewed by AI
- In 7-13B open vision-language models, object hallucination is not explained by weak visual attention e hallucinated and real object tokens draw comparable attention to specific image regions in mid-to-late layers; the separating signal is whether a logit-lens decode of those attended visual tokens matches the generated object word.Grounding answers in the image · reviewed by AI
- In 7-8B multimodal models, object hallucination during visually grounded token generation tracks a shift of the visual-language information ratio toward language inside mixed-modality "synergy" attention heads rather than a shortage of visual-specific heads, and rescaling visual versus language value vectors in those heads causally moves hallucination in both directions.Grounding answers in the image · reviewed by AI
- For binary visual question answering with open LVLMs (InstructBLIP-7B, Qwen2.5-VL, InternVL-4B), a hallucinated answer can be held with high response consistency, so a single consistency/uncertainty score misses it; separating consistency into four probe groups (image vs. text perturbation crossed with affirmative vs. negated statement) and classifying on the joint within- and between-group agreement pattern detects hallucination better than any single-score black-box or white-box baseline.Stating false facts confidently · reviewed by AI
- For training-free inference-time hallucination mitigation in 7B vision- language models, reported gains on object-hallucination benchmarks are largely inseparable from reduced informativeness: hallucination rate and object recall/coverage move together, so a lower score can mean the model mentioned fewer visual entities rather than grounded better.Whether the measurement made the finding · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Show that varying visual encoder quality leaves object-existence hallucination rates unchanged, or that the image-text similarity and attention-entropy differences between hallucinated and correct cases disappear under matched question difficulty. Proposed technique, not catalogued: Adversarial contrastive fine-tuning on hallucination-flipped image pairs.