7b-class-lvlms-llava-1-5-minigpt-4-mplug-owl2-rescore-object-tokens CLM-0535In 7B-class LVLMs (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2) that rescore object tokens during decoding with a frozen CLIP image-text similarity, the CLIP signal must be weighted by the model's own entropy over the object candidates: a fixed mixing coefficient is sensitive to its value and gives smaller hallucination reductions, and letting CLIP fully override the LVLM logits degrades output.
Capability: Grounding answers in the image
Observed on
Object hallucination in open-ended captioning; frozen CLIP ViT-B/32 validator, object tokens picked from the top-k pool via WordNet; does not apply to tasks without explicit object.
Sources
- Validated by: CHAIR_S/CHAIR_I and BLEU on 500 MSCOCO images over three seeds, OPOPE accuracy/precision/F1, and GPT-4V scores on LLaVA-Bench, against greedy, beam, DoLa, OPERA, VCD, HALC, Woodpecker, LURE and Nullu; reported CHAIR_S 14.27 vs 15.20 (Nullu) and 20.40 (greedy) on LLaVA-1.5, with throughput 0.59 vs 0.72 items/s for greedy; ablations sweep the fixed coefficient and the candidate-pool size (best at 10).. Measured on three LVLMs and three benchmarks with seed variance reported, plus a coefficient-sweep ablation on LLaVA-1.5 only. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In LVLMs asked to describe people (LLaVA, InstructBLIP, Shikra), masking attention so that text tokens read visual tokens but language-only self-attention is suppressed in late decoder layers lowers gender misattribution while keeping answer coverage high, whereas visual-only or language-only masking scores well on the bias metric only because the model names a gender in under 10% of cases.Grounding answers in the image · reviewed by AI
- In 4-7B open vision-language models (LLaVA-1.5-7B, Qwen2.5-VL-7B, InternVL-2.5-4B), how the cross-modal attention distribution over visual tokens shifts between adjacent decoder layers separates hallucinated from grounded object predictions better than the model's sensitivity to masking the attended region, and better than attention drift measured against a fixed first or last layer anchor.Grounding answers in the image · reviewed by AI
- In 7B-32B vision-language models, applying a bounded decoding-time logit penalty to candidate tokens whose rank-pooled visual-state evidence falls below a per-token reference estimated offline from unlabeled images lowers object-hallucination rates (CHAIR_I down up to 21.3% on four 7B backbones) while largely preserving descriptive and general multimodal scores, at about 1.33% added per-token latency once image evidence is cached at prefill.Grounding answers in the image · reviewed by AI
- For small open-weight Qwen2.5 models (1.5B-3B) doing multi-step math reasoning, features derived from the spread of attention entropy across layers at each generation step predict whether the final answer is wrong better than output-logit confidence features, with the margin larger on harder problems (+0.0093 AUC on GSM8K, +0.0759 on MATH-500 at 3B).Stating false facts confidently · reviewed by AI
- In 7B-class vision-language models (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2), object hallucination is carried by an anisotropic set of directions in deep-layer hidden states, so attenuating each eigenmode in proportion to its hallucination-to-truthful variance ratio — estimated once offline from paired truthful/hallucinated captions and folded into the FFN output projection — lowers CHAIR rates, while a mean-offset subtraction or isotropic shrinkage of the same representations does not.Grounding answers in the image · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A sweep showing some fixed mixing coefficient matches or beats entropy-adaptive weighting on CHAIR_S/CHAIR_I across these backbones, or showing the entropy weight's gain disappears once coverage/informativeness is held constant.