7-8b-vision-language-models-fine-tuning-visual-text-embeddings-toward-perturbation-averaged CLM-0004In 7-8B vision-language models, fine-tuning visual and text embeddings toward perturbation-averaged and ground-truth anchors before cross-modal contrastive alignment lowers object and attribute hallucination rates on captioning and existence benchmarks without reducing object coverage or general VQA accuracy.
Capability: Grounding answers in the image
Observed on
Single-image, single-turn settings at 7-8B scale (LLaVA-1.5, LLaVA-1.6, Qwen3-VL-8B); 2K training pairs drawn from RLHF-V chosen/rejected responses; evaluated on AMBER, ObjHal, MMH.
Sources
- supportsHallucination Mitigation for Large Vision-Language Models via Implicit Feature StabilizationMeasured on three backbones against re-run base models and re-run DPO and CHiP under matched data; AMBER CHAIR falls 7.8->4.2, 8.3->3.1, 5.9->2.9, with coverage rising and VQA-v2/TextVQA matched or improved. Ablations separate the intra-modal stage from the alignment stage. Not isolated: no ablation against a data-matched plain SFT baseline, and some comparison rows are literature-reported rather than controlled.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In 7B open vision-language models (LLaVA-1.5, Shikra), the per-token log-probability gap between a normal forward pass and one with visual features zeroed shrinks as a caption gets longer, and while that gap ranks hallucinated tokens slightly better than entropy or max-probability, its absolute separation is weak (AUROC around 0.57-0.59) and it fails entirely when aggregated to sentence level.Grounding answers in the image · reviewed by AI
- For training-free inference-time hallucination mitigation in 7B vision- language models, reported gains on object-hallucination benchmarks are largely inseparable from reduced informativeness: hallucination rate and object recall/coverage move together, so a lower score can mean the model mentioned fewer visual entities rather than grounded better.Whether the measurement made the finding · reviewed by AI
- In 7B vision-language models that prune visual tokens during decoding for efficiency, hallucination rises as the generated caption gets longer, because pruning is irreversible and discards fine-grained evidence needed at later decoding steps; keeping the pruned tokens as a retrievable bank and pulling them back when next-token entropy spikes recovers the grounding without a second forward pass.Grounding answers in the image · reviewed by AI
- In 7B-32B vision-language models, applying a bounded decoding-time logit penalty to candidate tokens whose rank-pooled visual-state evidence falls below a per-token reference estimated offline from unlabeled images lowers object-hallucination rates (CHAIR_I down up to 21.3% on four 7B backbones) while largely preserving descriptive and general multimodal scores, at about 1.33% added per-token latency once image evidence is cached at prefill.Grounding answers in the image · reviewed by AI
- For detecting hallucinated text spans in vision-language model outputs, span localization and confidence calibration come from different architectures: a fine-tuned generative VLM proposes better span boundaries but scores them poorly, while a discriminative token tagger trained with a direct regression loss on human-consensus confidence gives better-correlated probabilities but conservative spans, so using the generative spans as proposals and the tagger's probabilities as scores beats either alone.Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Replication on the same backbones showing that hallucination reductions are matched by drops in object coverage or in VQA-v2/TextVQA accuracy, or that a plain generation-loss fine-tune on the same 2K data gives the same hallucination reduction without the stabilization objectives. Proposed technique, not catalogued: Perturbation-invariance fine-tuning for vision-language embeddings.