7-13b-multimodal-models-verifying-each-object-claim-frozen-open-vocabulary CLM-0343For 7-13B multimodal models, verifying each object claim with a frozen open-vocabulary detector lowers object hallucination only if evidence is pooled across semantically equivalent phrasings and checked for spatial agreement (the preferred boxes of different queries overlapping on one instance); a single-query maximum score is a weaker signal, and shuffling the spatial evidence across samples drives its discrimination to chance.
Capability: Grounding answers in the image
Observed on
LLaVA-1.5-7B, mPLUG-Owl2, MiniGPT-4 Vicuna-13B with a frozen OWL-ViT-base verifier; POPE (COCO/A-OKVQA/GQA), MME-Existence, and CHAIR on 500 COCO images; claims reducible to object.
Model cards: GPT-4
Sources
- Validated by: On LLaVA-1.5-7B, POPE accuracy averaged over three sources rose 1.81 (Popular) and 3.17 (Adversarial) points over vanilla, CHAIRs fell 49.40 to 32.80 and CHAIRi 13.82 to 8.22; a matched-random-deletion control removing the same number of claims only reached CHAIRs 43.40; ablations show single-query 80.20 vs full 81.97 GQA-Adversarial accuracy, and shuffling QIRV evidence drops ROC-AUC 63.32 to 50.43. Random-sampling POPE accuracy fell (90.33 to 88.54) and recall drops as calibration strength grows.. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For training-free inference-time hallucination interventions in 7B-class vision-language models, increasing intervention strength (finer regional evidence partitioning, longer attention-routing windows, stronger textual perturbation) keeps lowering CHAIR object-hallucination scores while producing empty or reduced-coverage responses, so lower hallucination numbers at high intervention strength partly reflect the model saying less rather than grounding better.Grounding answers in the image · reviewed by AI
- For training-free inference-time hallucination mitigation in 7B vision- language models, reported gains on object-hallucination benchmarks are largely inseparable from reduced informativeness: hallucination rate and object recall/coverage move together, so a lower score can mean the model mentioned fewer visual entities rather than grounded better.Whether the measurement made the finding · reviewed by AI
- In frozen large vision-language models, token-level hallucination evidence is distributed across the relational, spatial and sequential structure of the model's internal trace rather than contained in any single token's hidden state, so a detector that reads the attention graph over image patches, query and response tokens with interleaved GNN/ViT/GRU modules localizes hallucinated spans better than flat probes on the same activations.Grounding answers in the image · reviewed by AI
- In open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.Grounding answers in the image · reviewed by AI
- In open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.Grounding answers in the image · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An ablation on the same models and benchmarks showing single-query detector verification at a tuned threshold matches full cross-query semantic-spatial verification on CHAIRi and POPE Adversarial accuracy at equal rejection budget, or spatial-evidence shuffling leaving discrimination intact. Drafted stance toward training-free-inference-time-hallucination-interventions-7b-class-vision-language-models-increasing-intervention: supports -- Raising the external-evidence calibration coefficient keeps correcting errors but cuts recall from 84.93% to 72.13% and lowers accuracy, so lower hallucination at high intervention strength again comes with less content committed.