7b-open-vision-language-models-llava-1-5-shikra-per-token-log-probability CLM-0349In 7B open vision-language models (LLaVA-1.5, Shikra), the per-token log-probability gap between a normal forward pass and one with visual features zeroed shrinks as a caption gets longer, and while that gap ranks hallucinated tokens slightly better than entropy or max-probability, its absolute separation is weak (AUROC around 0.57-0.59) and it fails entirely when aggregated to sentence level.
Capability: Grounding answers in the image
Observed on
7B LVLMs with a separable visual encoder; InstructBLIP's Q-Former gives a near-zero gap, so the signal does not exist in all architectures. Measured on COCO captioning with CHAIR l.
Sources
- Validated by: Compared against Sample, Beam, OPERA, VCD, AGLA, SID and ICD on five benchmarks with LLaVA-1.5, InstructBLIP, Shikra. Large gain only on MMHalBench (3.55 overall vs 1.69 for OPERA, hallucination ratio 0.44 vs 0.74-0.83, single GPT judge over 96 samples); roughly tied or worse on HallusionBench, CHAIR sentence level, and MMMU. Outputs are shorter (95.3 vs 101.8 tokens), and length-normalized CHAIRs margins are small and reverse on Shikra.. Measured: Delta trajectories on 30 COCO images (LLaVA 0.39 to 0.07, Shikra 1.13 to 0.41); [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In 7B vision-language models that prune visual tokens during decoding for efficiency, hallucination rises as the generated caption gets longer, because pruning is irreversible and discards fine-grained evidence needed at later decoding steps; keeping the pruned tokens as a retrievable bank and pulling them back when next-token entropy spikes recovers the grounding without a second forward pass.Grounding answers in the image · reviewed by AI
- In open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.Grounding answers in the image · reviewed by AI
- In open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.Grounding answers in the image · reviewed by AI
- In 7B-32B vision-language models, evidence for a candidate token read out of visual-token hidden states cannot be pooled as probabilities across visual positions, because each position's distribution sharpness differs (the rank-10 probability spans roughly 8-14x across positions and affine rescaling does not remove it); pooling vocabulary ranks instead is invariant to that scale, and because most of the remaining variability is between tokens rather than between images, the pooled rank must be compared against a per-token reference rather than one global threshold.Grounding answers in the image · reviewed by AI
- In 7-13B open vision-language models, object hallucination is not explained by weak visual attention e hallucinated and real object tokens draw comparable attention to specific image regions in mid-to-late layers; the separating signal is whether a logit-lens decode of those attended visual tokens matches the generated object word.Grounding answers in the image · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Showing that the vision-ablation log-prob gap stays flat or rises across decoding steps in these models, or that it separates hallucinated from grounded tokens at high AUROC including at sentence level. Drafted stance toward training-free-inference-time-hallucination-interventions-7b-class-vision-language-models-increasing-intervention: supports -- The intervention lowers hallucination metrics while also shortening captions and lowering total token count, and the authors state they have not disentangled the two.