lvlms-asked-describe-people-llava-instructblip-shikra-masking-attention CLM-0435In LVLMs asked to describe people (LLaVA, InstructBLIP, Shikra), masking attention so that text tokens read visual tokens but language-only self-attention is suppressed in late decoder layers lowers gender misattribution while keeping answer coverage high, whereas visual-only or language-only masking scores well on the bias metric only because the model names a gender in under 10% of cases.
Capability: Grounding answers in the image
Observed on
Training-free inference-time masking on three open LVLMs; gender attribution judged by keyword matching on FACET, MS COCO and FairFace, with outputs that name no gender excluded fr.
Sources
- Validated by: Gender bias score on FACET (0.9274 to 0.9928 for LLaVA), MS COCO (0.6708 to 0.9978), FairFace (0.5988 to 0.8782 for Shikra), against six prior debiasing baselines; POPE accuracy and MMMU score held near baseline (LLaVA 82.63 to 83.06 POPE, 32.4 to 31.8 MMMU); coverage per attention mode reported (V2L 95.8% vs visual-only 9.3%, language-only 4.9%, L2V 0.2%).. Measured on three LVLMs and three person-attribute benchmarks with per-mode coverage reported, plus POPE/MMMU capability checks and a layer-group grid search. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In 4-7B open vision-language models (LLaVA-1.5-7B, Qwen2.5-VL-7B, InternVL-2.5-4B), how the cross-modal attention distribution over visual tokens shifts between adjacent decoder layers separates hallucinated from grounded object predictions better than the model's sensitivity to masking the attended region, and better than attention drift measured against a fixed first or last layer anchor.Grounding answers in the image · reviewed by AI
- In masked diffusion language models (LLaDA, Dream) asked to generate around a frozen constraint span placed mid-output, failures to honor the constraint come partly from too little attention being allocated to that span during denoising: adding a fixed additive bias to attention logits on the span raises constraint adherence and overall quality up to a moderate bias, then degrades fluency and contextual fit as the bias grows.Following an unfamiliar procedure · reviewed by AI
- For binary visual question answering with open LVLMs (InstructBLIP-7B, Qwen2.5-VL, InternVL-4B), a hallucinated answer can be held with high response consistency, so a single consistency/uncertainty score misses it; separating consistency into four probe groups (image vs. text perturbation crossed with affirmative vs. negated statement) and classifying on the joint within- and between-group agreement pattern detects hallucination better than any single-score black-box or white-box baseline.Stating false facts confidently · reviewed by AI
- In LLaVA-1.5-7B captioning, restricting a LoRA adapter to attention heads picked by a hallucination diagnosis (attention-to-image drop around hallucinated object words, then ablation screening) reduces object hallucination, while an identically trained layer-matched random-head adapter does not — so the selection, not the added adapter capacity, carries the effect.Grounding answers in the image · reviewed by AI
- In 7B-32B vision-language models, evidence for a candidate token read out of visual-token hidden states cannot be pooled as probabilities across visual positions, because each position's distribution sharpness differs (the rank-10 probability spans roughly 8-14x across positions and affine rescaling does not remove it); pooling vocabulary ranks instead is invariant to that scale, and because most of the remaining variability is between tokens rather than between images, the pooled rank must be compared against a per-token reference rather than one global threshold.Grounding answers in the image · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: An LVLM where this masking scheme leaves gender misattribution unchanged at matched coverage, or where the reported gains disappear once coverage is held fixed across modes. Drafted stance toward hallucination-metric-unsupported-claim-rate-computed-over-whatever-claims-model: supports -- A bias score computed only over outputs that commit to a gender rewards evasion: two attention modes reach near-perfect scores at under 10% coverage.