7b-32b-vision-language-models-applying-bounded-decoding-time-logit-penalty-candidate CLM-0533In 7B-32B vision-language models, applying a bounded decoding-time logit penalty to candidate tokens whose rank-pooled visual-state evidence falls below a per-token reference estimated offline from unlabeled images lowers object-hallucination rates (CHAIR_I down up to 21.3% on four 7B backbones) while largely preserving descriptive and general multimodal scores, at about 1.33% added per-token latency once image evidence is cached at prefill.
Capability: Grounding answers in the image
Observed on
Open vision-language models with accessible visual-token hidden states and a logit lens readout; requires a one-time offline calibration pass per model on unlabeled images..
Sources
- Measured: hallucination metric reductions across 11 models and six families up to 32B, plus latency. The paper reports descriptive and general performance 'largely preserved or improved' but the excerpt gives no coverage/recall numbers, so the known confound in this literature (lower hallucination via saying less) is not isolated here.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In 7B-32B vision-language models, evidence for a candidate token read out of visual-token hidden states cannot be pooled as probabilities across visual positions, because each position's distribution sharpness differs (the rank-10 probability spans roughly 8-14x across positions and affine rescaling does not remove it); pooling vocabulary ranks instead is invariant to that scale, and because most of the remaining variability is between tokens rather than between images, the pooled rank must be compared against a per-token reference rather than one global threshold.Grounding answers in the image · reviewed by AI
- In 7B vision-language models that prune visual tokens during decoding for efficiency, hallucination rises as the generated caption gets longer, because pruning is irreversible and discards fine-grained evidence needed at later decoding steps; keeping the pruned tokens as a retrievable bank and pulling them back when next-token entropy spikes recovers the grounding without a second forward pass.Grounding answers in the image · reviewed by AI
- In 7B open vision-language models (LLaVA-1.5, Shikra), the per-token log-probability gap between a normal forward pass and one with visual features zeroed shrinks as a caption gets longer, and while that gap ranks hallucinated tokens slightly better than entropy or max-probability, its absolute separation is weak (AUROC around 0.57-0.59) and it fails entirely when aggregated to sentence level.Grounding answers in the image · reviewed by AI
- In 7-8B vision-language models, fine-tuning visual and text embeddings toward perturbation-averaged and ground-truth anchors before cross-modal contrastive alignment lowers object and attribute hallucination rates on captioning and existence benchmarks without reducing object coverage or general VQA accuracy.Grounding answers in the image · reviewed by AI
- In 7B-class LVLMs (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2), hallucination directions in activation space are not a single mode: clustering hallucinated-minus-faithful hidden-state differences into several low-rank subspaces and weighting them per test image lowers object-hallucination rates slightly below a single global null-space projection, with the margin small (CHAIR_I 5.30 to 4.92 on LLaVA-1.5) and inconsistent on yes/no existence probing.Grounding answers in the image · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A replication where the rank-plus-per-token-reference penalty fails to lower CHAIR_I relative to unmodified decoding, or where the reduction is accounted for by shorter or less detailed captions (lower object coverage) rather than better grounding. Drafted stance toward 7b-32b-vision-language-models-evidence-candidate-token-read-out-visual-token: supports -- The paper measures the across-position probability spread (13.8x for LLaVA-1.5-7B, 7.8x for Qwen2.5-VL-7B, only reduced to 11.0x and 4.2x by affine equalization) and attributes 66.4% of DMRR variability to between-token differences, motivating rank pooling against a per-token reference.