4-7b-open-vision-language-models-llava-1-5-7b-qwen2-5-vl-7b-internvl-2 CLM-0527observationsingle paperReviewed by AI
In 4-7B open vision-language models (LLaVA-1.5-7B, Qwen2.5-VL-7B, InternVL-2.5-4B), how the cross-modal attention distribution over visual tokens shifts between adjacent decoder layers separates hallucinated from grounded object predictions better than the model's sensitivity to masking the attended region, and better than attention drift measured against a fixed first or last layer anchor.
Capability: Grounding answers in the image
Observed on
Open-weight 4-7B LVLMs, object/attribute/relation hallucination detection on POPE, Pascal VOC, COCO captions and AMBER, with a small supervised MLP detector trained on the extracte.
Sources
- Validated by: ACC and AUROC against six detection baselines (IC, GLSim, NLL, Entropy, SVAR, DHCP) on three models and three datasets: best in 16 of 18 cells, with COCO-Caption AUROC gains of 5.6/8.4/4.6 points over the best baseline; ablations isolate each component (LLaVA drift-only AUROC 85.7, masking-only 72.8, combined 87.4) and compare masking-layer and masking-region choices.. Measured on three open LVLMs and four benchmarks with repeated runs and reported standard deviations; ablations cover drift metric, anchor formulation, masking layer and masking region. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In frozen large vision-language models, token-level hallucination evidence is distributed across the relational, spatial and sequential structure of the model's internal trace rather than contained in any single token's hidden state, so a detector that reads the attention graph over image patches, query and response tokens with interleaved GNN/ViT/GRU modules localizes hallucinated spans better than flat probes on the same activations.Grounding answers in the image · reviewed by AI
- In 7B-32B vision-language models, evidence for a candidate token read out of visual-token hidden states cannot be pooled as probabilities across visual positions, because each position's distribution sharpness differs (the rank-10 probability spans roughly 8-14x across positions and affine rescaling does not remove it); pooling vocabulary ranks instead is invariant to that scale, and because most of the remaining variability is between tokens rather than between images, the pooled rank must be compared against a per-token reference rather than one global threshold.Grounding answers in the image · reviewed by AI
- In open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.Grounding answers in the image · reviewed by AI
- In 7B-class vision-language models (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2), object hallucination is carried by an anisotropic set of directions in deep-layer hidden states, so attenuating each eigenmode in proportion to its hallucination-to-truthful variance ratio — estimated once offline from paired truthful/hallucinated captions and folded into the FFN output projection — lowers CHAIR rates, while a mean-offset subtraction or isotropic shrinkage of the same representations does not.Grounding answers in the image · reviewed by AI
- In 7-8B multimodal models, object hallucination during visually grounded token generation tracks a shift of the visual-language information ratio toward language inside mixed-modality "synergy" attention heads rather than a shortage of visual-specific heads, and rescaling visual versus language value vectors in those heads causally moves hallucination in both directions.Grounding answers in the image · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: On the same models and benchmarks, a masking-based or fixed-anchor drift feature matching or beating adjacent-layer drift AUROC, or the combined detector failing to beat drift-only features.