7b-class-lvlms-llava-1-5-minigpt-4-mplug-owl2-hallucination-directions-activation CLM-0534In 7B-class LVLMs (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2), hallucination directions in activation space are not a single mode: clustering hallucinated-minus-faithful hidden-state differences into several low-rank subspaces and weighting them per test image lowers object-hallucination rates slightly below a single global null-space projection, with the margin small (CHAIR_I 5.30 to 4.92 on LLaVA-1.5) and inconsistent on yes/no existence probing.
Capability: Grounding answers in the image
Observed on
Training-free editing of mid-layer hidden states in three open LVLMs, evaluated on CHAIR and offline POPE over MSCOCO val; gains over the single-subspace baseline are under one poi.
Sources
- Validated by: CHAIR_S/CHAIR_I/BLEU and offline-POPE accuracy/precision/F on MSCOCO across three LVLMs, mean and standard deviation over ten clustering runs, against greedy, beam, DoLa, OPERA, VCD, Woodpecker, LURE, HALC and the single-subspace Nullu baseline; ablations on cluster count, basis rank, and masking vs Gaussian noise vs blur.. Measured on two benchmarks, three LVLMs, ten runs with reported standard deviations. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For training-free inference-time hallucination interventions in 7B-class vision-language models, increasing intervention strength (finer regional evidence partitioning, longer attention-routing windows, stronger textual perturbation) keeps lowering CHAIR object-hallucination scores while producing empty or reduced-coverage responses, so lower hallucination numbers at high intervention strength partly reflect the model saying less rather than grounding better.Grounding answers in the image · reviewed by AI
- For training-free inference-time hallucination mitigation in 7B vision- language models, reported gains on object-hallucination benchmarks are largely inseparable from reduced informativeness: hallucination rate and object recall/coverage move together, so a lower score can mean the model mentioned fewer visual entities rather than grounded better.Whether the measurement made the finding · reviewed by AI
- In open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.Grounding answers in the image · reviewed by AI
- In open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.Grounding answers in the image · reviewed by AI
- In 7B-class vision-language models (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2), object hallucination is carried by an anisotropic set of directions in deep-layer hidden states, so attenuating each eigenmode in proportion to its hallucination-to-truthful variance ratio — estimated once offline from paired truthful/hallucinated captions and folded into the FFN output projection — lowers CHAIR rates, while a mean-offset subtraction or isotropic shrinkage of the same representations does not.Grounding answers in the image · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A matched comparison in which a single global subspace with the same total rank and the same masked-image probe matches the per-instance mixture on CHAIR and POPE, or in which randomly partitioned clusters work as well as the difference-vector clustering. Drafted stance toward training-free-inference-time-hallucination-interventions-7b-class-vision-language-models-increasing-intervention: supports -- The basis-rank ablation shows CHAIR_S falling monotonically from 16.2 to 11.3 as rank grows from 1 to 64 while BLEU falls from 15.6 to 12.7, which the authors read as loss of grounded content, matching the claim that stronger intervention buys lower hallucination scores with reduced informativeness.