7-8b-multimodal-models-object-hallucination-during-visually-grounded-token CLM-0344In 7-8B multimodal models, object hallucination during visually grounded token generation tracks a shift of the visual-language information ratio toward language inside mixed-modality "synergy" attention heads rather than a shortage of visual-specific heads, and rescaling visual versus language value vectors in those heads causally moves hallucination in both directions.
Capability: Grounding answers in the image
Observed on
7-8B open multimodal models (LLaVA-1.5/NeXT, Qwen2-2.5-3-VL, InternVL) with accessible internals; inference-time intervention with a per-model equilibrium factor; does not help whe.
Sources
- Validated by: POPE F1 87.7 and accuracy 88.3 and MME total 669.8 on LLaVA-1.5-7B versus 85.4/84.0/565.3 beam-search baseline and above ~20 published decoding and attention baselines; CHAIR_S 51.0 to 36.9 and CHAIR_I 15.2 to 10.7 with recall and output length maintained; gains repeated on five other backbones; head assignments 92-95% stable across four masking strategies.. Measured on POPE, CHAIR, MME, LLaVA-Bench and MMHal across seven MLLMs, with bidirectional causal intervention and robustness checks over masking strategies and thresholds. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In 7-13B open vision-language models, object hallucination is not explained by weak visual attention e hallucinated and real object tokens draw comparable attention to specific image regions in mid-to-late layers; the separating signal is whether a logit-lens decode of those attended visual tokens matches the generated object word.Grounding answers in the image · reviewed by AI
- In 7B-32B vision-language models, evidence for a candidate token read out of visual-token hidden states cannot be pooled as probabilities across visual positions, because each position's distribution sharpness differs (the rank-10 probability spans roughly 8-14x across positions and affine rescaling does not remove it); pooling vocabulary ranks instead is invariant to that scale, and because most of the remaining variability is between tokens rather than between images, the pooled rank must be compared against a per-token reference rather than one global threshold.Grounding answers in the image · reviewed by AI
- For short-answer object-existence questions in open multimodal models (LLaVA v1.5, MiniGPT-4, Qwen2.5-VL), object hallucination is driven partly by the visual encoder itself rather than only by language priors: hallucinated cases show lower image-text embedding similarity and inverted attention, and swapping in a weaker or noisier visual encoder lowers POPE accuracy while a stronger one raises it.Grounding answers in the image · reviewed by AI
- For training-free inference-time hallucination mitigation in 7B vision- language models, reported gains on object-hallucination benchmarks are largely inseparable from reduced informativeness: hallucination rate and object recall/coverage move together, so a lower score can mean the model mentioned fewer visual entities rather than grounded better.Whether the measurement made the finding · reviewed by AI
- In frozen large vision-language models, token-level hallucination evidence is distributed across the relational, spatial and sequential structure of the model's internal trace rather than contained in any single token's hidden state, so a detector that reads the attention graph over image patches, query and response tokens with interleaved GNN/ViT/GRU modules localizes hallucinated spans better than flat probes on the same activations.Grounding answers in the image · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Show that hallucinated tokens do not differ from correct tokens in the visual-language ratio of synergy heads once head classification is done properly, or that lowering the visual share in synergy heads does not induce hallucination while lowering it in modality-specific heads does; or that the CHAIR gain comes from reduced object coverage.