7-8b-vision-language-models-fine-tuning-visual-text-embeddings-toward-perturbation-averagedIn 7-8B vision-language models, fine-tuning visual and text embeddings toward perturbation-averaged and ground-truth anchors before cross-modal contrastive alignment lowers object and attribute hallucination rates on captioning and existence benchmarks without reducing object coverage or general VQA accuracy.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Grounding answers in the image
Observed on
Single-image, single-turn settings at 7-8B scale (LLaVA-1.5, LLaVA-1.6, Qwen3-VL-8B); 2K training pairs drawn from RLHF-V chosen/rejected responses; evaluated on AMBER, ObjHal, MMH.
Sources
- supportsHallucination Mitigation for Large Vision-Language Models via Implicit Feature StabilizationMeasured on three backbones against re-run base models and re-run DPO and CHiP under matched data; AMBER CHAIR falls 7.8->4.2, 8.3->3.1, 5.9->2.9, with coverage rising and VQA-v2/TextVQA matched or improved. Ablations separate the intra-modal stage from the alignment stage. Not isolated: no ablation against a data-matched plain SFT baseline, and some comparison rows are literature-reported rather than controlled.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- For training-free inference-time hallucination mitigation in 7B vision- language models, reported gains on object-hallucination benchmarks are largely inseparable from reduced informativeness: hallucination rate and object recall/coverage move together, so a lower score can mean the model mentioned fewer visual entities rather than grounded better.Whether the measurement made the finding · unreviewed
- In 7B vision-language models that prune visual tokens during decoding for efficiency, hallucination rises as the generated caption gets longer, because pruning is irreversible and discards fine-grained evidence needed at later decoding steps; keeping the pruned tokens as a retrievable bank and pulling them back when next-token entropy spikes recovers the grounding without a second forward pass.Grounding answers in the image · unreviewed
- For detecting hallucinated text spans in vision-language model outputs, span localization and confidence calibration come from different architectures: a fine-tuned generative VLM proposes better span boundaries but scores them poorly, while a discriminative token tagger trained with a direct regression loss on human-consensus confidence gives better-correlated probabilities but conservative spans, so using the generative spans as proposals and the tagger's probabilities as scores beats either alone.Stating false facts confidently · unreviewed
- In open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.Grounding answers in the image · unreviewed
- For short-answer object-existence questions in open multimodal models (LLaVA v1.5, MiniGPT-4, Qwen2.5-VL), object hallucination is driven partly by the visual encoder itself rather than only by language priors: hallucinated cases show lower image-text embedding similarity and inverted attention, and swapping in a weaker or noisier visual encoder lowers POPE accuracy while a stronger one raises it.Grounding answers in the image · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Replication on the same backbones showing that hallucination reductions are matched by drops in object coverage or in VQA-v2/TextVQA accuracy, or that a plain generation-loss fine-tune on the same 2K data gives the same hallucination reduction without the stabilization objectives. Proposed technique, not catalogued: Perturbation-invariance fine-tuning for vision-language embeddings.