llava-1-5-7b-captioning-restricting-lora-adapter-attention-heads-picked CLM-0239In LLaVA-1.5-7B captioning, restricting a LoRA adapter to attention heads picked by a hallucination diagnosis (attention-to-image drop around hallucinated object words, then ablation screening) reduces object hallucination, while an identically trained layer-matched random-head adapter does not — so the selection, not the added adapter capacity, carries the effect.
Capability: Grounding answers in the image
Observed on
One model (LLaVA-1.5-7B), one captioning prompt, COCO val2014 images, CHAIR scoring; 32 selected heads across 19 layers; control split of 200 images. Gains come with shorter captio.
Sources
- Validated by: Not stated. Measured on 400 held-out COCO images (CHAIRs 0.370 to 0.230, CHAIRi 0.156 to 0.096, paired sign-flip p<0.001) plus a 200-image control split where random-head LoRA gave CHAIRs 0.400 vs 0.100 for selected heads. Fixed-budget runs (64/80/128 tokens) rule out pure truncation, but no fully length-matched or all-head-LoRA baseline, and recall falls, so precision-coverage tradeoff is not isolated.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In LLaVA-1.5-7B captioning, restricting a LoRA adapter to attention heads picked by a hallucination diagnosis (attention-to-image drop around hallucinated object words, then ablation screening) reduces object hallucination, while an identically trained layer-matched random-head adapter does not — so the selection, not the added adapter capacity, carries the effect.Grounding answers in the image · reviewed by AI
- In 7B-class vision-language models (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2), object hallucination is carried by an anisotropic set of directions in deep-layer hidden states, so attenuating each eigenmode in proportion to its hallucination-to-truthful variance ratio — estimated once offline from paired truthful/hallucinated captions and folded into the FFN output projection — lowers CHAIR rates, while a mean-offset subtraction or isotropic shrinkage of the same representations does not.Grounding answers in the image · reviewed by AI
- In 7B-class LVLMs (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2), hallucination directions in activation space are not a single mode: clustering hallucinated-minus-faithful hidden-state differences into several low-rank subspaces and weighting them per test image lowers object-hallucination rates slightly below a single global null-space projection, with the margin small (CHAIR_I 5.30 to 4.92 on LLaVA-1.5) and inconsistent on yes/no existence probing.Grounding answers in the image · reviewed by AI
- When adding image-level preference pairs to multimodal DPO for LLaVA-1.5 7B/13B, constructing the rejected image by selecting, among many equal-size random masks, the variant with lowest CLIP similarity to the original lowers object-hallucination rates more than coarse alternatives (blockwise masking, rotation, cropping, or a fully blacked-out image), at the cost of slightly lower object coverage.Grounding answers in the image · reviewed by AI
- When Video-LLMs are asked to verify sentence by sentence whether a dense video caption is supported by the video, they detect hallucinated sentences less often in captions they generated themselves than in captions from other models, and detection of temporal errors (action, order, causality, frequency, state change) improves much less with model scale than detection of static-frame errors.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A random-head or all-head LoRA trained with the same DPO pairs and matched length reaching the same CHAIR reduction, or the selected heads failing to transfer to other VLMs/prompts.