7b-vision-language-models-prune-visual-tokens-during-decoding-efficiency CLM-0005In 7B vision-language models that prune visual tokens during decoding for efficiency, hallucination rises as the generated caption gets longer, because pruning is irreversible and discards fine-grained evidence needed at later decoding steps; keeping the pruned tokens as a retrievable bank and pulling them back when next-token entropy spikes recovers the grounding without a second forward pass.
Capability: Grounding answers in the image
Observed on
7B LLaVA-1.5, LLaVA-NEXT and Qwen-VL with training-free decoding-time sparsification; open-ended captioning benchmarks (CHAIR, AMBER) and discriminative probes (POPE, MME)..
Sources
- Measured across five benchmarks and three 7B backbones against training-free baselines (VCD, M3ID, VTI, AvisC); component ablation on MME shows removing the retrieval module causes the largest drop. Length-dependence of sparse-decoder hallucination is shown only via one figure on one prior method (VASparse). Caption Recall falls on LLaVA-1.5 (77.3 to 74.4) while CHAIR improves, so part of the gain there may be reduced informativeness; on LLaVA-NEXT recall rises.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In 7B open vision-language models (LLaVA-1.5, Shikra), the per-token log-probability gap between a normal forward pass and one with visual features zeroed shrinks as a caption gets longer, and while that gap ranks hallucinated tokens slightly better than entropy or max-probability, its absolute separation is weak (AUROC around 0.57-0.59) and it fails entirely when aggregated to sentence level.Grounding answers in the image · reviewed by AI
- In 7B-32B vision-language models, applying a bounded decoding-time logit penalty to candidate tokens whose rank-pooled visual-state evidence falls below a per-token reference estimated offline from unlabeled images lowers object-hallucination rates (CHAIR_I down up to 21.3% on four 7B backbones) while largely preserving descriptive and general multimodal scores, at about 1.33% added per-token latency once image evidence is cached at prefill.Grounding answers in the image · reviewed by AI
- In 7B-32B vision-language models, evidence for a candidate token read out of visual-token hidden states cannot be pooled as probabilities across visual positions, because each position's distribution sharpness differs (the rank-10 probability spans roughly 8-14x across positions and affine rescaling does not remove it); pooling vocabulary ranks instead is invariant to that scale, and because most of the remaining variability is between tokens rather than between images, the pooled rank must be compared against a per-token reference rather than one global threshold.Grounding answers in the image · reviewed by AI
- In 7-13B open vision-language models, object hallucination is not explained by weak visual attention e hallucinated and real object tokens draw comparable attention to specific image regions in mid-to-late layers; the separating signal is whether a logit-lens decode of those attended visual tokens matches the generated object word.Grounding answers in the image · reviewed by AI
- In diffusion vision-language models generating free-form captions, object tokens that are hallucinated commit at later denoising steps and with lower per-token confidence than grounded object tokens, so the commit-step trajectory is an available internal hallucination signal that has no counterpart in autoregressive decoding.Grounding answers in the image · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Show that a sparse decoder's hallucination rate does not grow with generation length, or that restoring pruned visual tokens on demand gives no CHAIR/HalRate improvement over the same sparsifier without restoration at matched caption length and recall. Proposed technique, not catalogued: Deferred visual token bank with uncertainty-triggered retrieval.