Grounding answers in the image

Claims

• observationsingle paper↑1↓0
  In 4-7B open vision-language models (LLaVA-1.5-7B, Qwen2.5-VL-7B, InternVL-2.5-4B), how the cross-modal attention distribution over visual tokens shifts between adjacent decoder layers separates hallucinated from grounded object predictions better than the model's sensitivity to masking the attended region, and better than attention drift measured against a fixed first or last layer anchor.
• mechanismsingle paper↑1↓0
  For 7-13B multimodal models, verifying each object claim with a frozen open-vocabulary detector lowers object hallucination only if evidence is pooled across semantically equivalent phrasings and checked for spatial agreement (the preferred boxes of different queries overlapping on one instance); a single-query maximum score is a weaker signal, and shuffling the spatial evidence across samples drives its discrimination to chance.
• mechanismsingle paper↑1↓0
  In 7-13B open vision-language models, object hallucination is not explained by weak visual attention e hallucinated and real object tokens draw comparable attention to specific image regions in mid-to-late layers; the separating signal is whether a logit-lens decode of those attended visual tokens matches the generated object word.
• mechanismsingle paper↑1↓0
  In 7-8B multimodal models, object hallucination during visually grounded token generation tracks a shift of the visual-language information ratio toward language inside mixed-modality "synergy" attention heads rather than a shortage of visual-specific heads, and rescaling visual versus language value vectors in those heads causally moves hallucination in both directions.
• observationsingle paper↑1↓0
  In 7-8B vision-language models, fine-tuning visual and text embeddings toward perturbation-averaged and ground-truth anchors before cross-modal contrastive alignment lowers object and attribute hallucination rates on captioning and existence benchmarks without reducing object coverage or general VQA accuracy.
• observationsingle paper↑1↓0
  In 7B-32B vision-language models, applying a bounded decoding-time logit penalty to candidate tokens whose rank-pooled visual-state evidence falls below a per-token reference estimated offline from unlabeled images lowers object-hallucination rates (CHAIR_I down up to 21.3% on four 7B backbones) while largely preserving descriptive and general multimodal scores, at about 1.33% added per-token latency once image evidence is cached at prefill.
• mechanismsingle paper↑1↓0
  In 7B-32B vision-language models, evidence for a candidate token read out of visual-token hidden states cannot be pooled as probabilities across visual positions, because each position's distribution sharpness differs (the rank-10 probability spans roughly 8-14x across positions and affine rescaling does not remove it); pooling vocabulary ranks instead is invariant to that scale, and because most of the remaining variability is between tokens rather than between images, the pooled rank must be compared against a per-token reference rather than one global threshold.
• mechanismsingle paper↑1↓0
  In 7B-class LVLMs (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2), hallucination directions in activation space are not a single mode: clustering hallucinated-minus-faithful hidden-state differences into several low-rank subspaces and weighting them per test image lowers object-hallucination rates slightly below a single global null-space projection, with the margin small (CHAIR_I 5.30 to 4.92 on LLaVA-1.5) and inconsistent on yes/no existence probing.
• mechanismsingle paper↑1↓0
  In 7B-class LVLMs (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2) that rescore object tokens during decoding with a frozen CLIP image-text similarity, the CLIP signal must be weighted by the model's own entropy over the object candidates: a fixed mixing coefficient is sensitive to its value and gives smaller hallucination reductions, and letting CLIP fully override the LVLM logits degrades output.
• mechanismsingle paper↑1↓0
  In 7B-class vision-language models (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2), object hallucination is carried by an anisotropic set of directions in deep-layer hidden states, so attenuating each eigenmode in proportion to its hallucination-to-truthful variance ratio — estimated once offline from paired truthful/hallucinated captions and folded into the FFN output projection — lowers CHAIR rates, while a mean-offset subtraction or isotropic shrinkage of the same representations does not.
• observationsingle paper↑1↓0
  In 7B open vision-language models (LLaVA-1.5, Shikra), the per-token log-probability gap between a normal forward pass and one with visual features zeroed shrinks as a caption gets longer, and while that gap ranks hallucinated tokens slightly better than entropy or max-probability, its absolute separation is weak (AUROC around 0.57-0.59) and it fails entirely when aggregated to sentence level.
• mechanismsingle paper↑1↓0
  In 7B vision-language models that prune visual tokens during decoding for efficiency, hallucination rises as the generated caption gets longer, because pruning is irreversible and discards fine-grained evidence needed at later decoding steps; keeping the pruned tokens as a retrievable bank and pulling them back when next-token entropy spikes recovers the grounding without a second forward pass.
• mechanismsingle paper↑1↓0
  For 8B multimodal models, preference tuning and plain outcome-reward GRPO lower hallucination rates while making stated confidence worse calibrated than the untuned backbone, because rank- or correctness-based rewards never score the numerical accuracy of the model's own confidence; adding a Brier-plus-group-ECE calibration term to the GRPO reward cuts calibration error sharply and leaves accuracy metrics largely unchanged.
• mechanismsingle paper↑1↓0
  When adding image-level preference pairs to multimodal DPO for LLaVA-1.5 7B/13B, constructing the rejected image by selecting, among many equal-size random masks, the variant with lowest CLIP similarity to the original lowers object-hallucination rates more than coarse alternatives (blockwise masking, rotation, cropping, or a fully blacked-out image), at the cost of slightly lower object coverage.
• mechanismsingle paper↑1↓0
  For detecting ungrounded object mentions in open-ended captions from white-box vision-language models, a single internal support signal (token likelihood, attention share, or image-text similarity) confuses support that comes from object-specific image evidence with support that comes from scene plausibility or the generated text prefix; scoring both whether image tokens carry object-specific evidence and whether support is more image-derived than prefix-derived separates these and raises detection AUROC.
• observationsingle paper↑1↓0
  In diffusion vision-language models generating free-form captions, object tokens that are hallucinated commit at later denoising steps and with lower per-token confidence than grounded object tokens, so the commit-step trajectory is an available internal hallucination signal that has no counterpart in autoregressive decoding.
• mechanismsingle paper↑1↓0
  In frozen large vision-language models, token-level hallucination evidence is distributed across the relational, spatial and sequential structure of the model's internal trace rather than contained in any single token's hidden state, so a detector that reads the attention graph over image patches, query and response tokens with interleaved GNN/ViT/GRU modules localizes hallucinated spans better than flat probes on the same activations.
• mechanismsingle paper↑1↓0
  In LLaVA-1.5-7B captioning, restricting a LoRA adapter to attention heads picked by a hallucination diagnosis (attention-to-image drop around hallucinated object words, then ablation screening) reduces object hallucination, while an identically trained layer-matched random-head adapter does not — so the selection, not the added adapter capacity, carries the effect.
• mechanismsingle paper↑1↓0
  In LLaVA-1.5-7B captioning, restricting a LoRA adapter to attention heads picked by a hallucination diagnosis (attention-to-image drop around hallucinated object words, then ablation screening) reduces object hallucination, while an identically trained layer-matched random-head adapter does not — so the selection, not the added adapter capacity, carries the effect.
• mechanismsingle paper↑1↓0
  In LVLMs asked to describe people (LLaVA, InstructBLIP, Shikra), masking attention so that text tokens read visual tokens but language-only self-attention is suppressed in late decoder layers lowers gender misattribution while keeping answer coverage high, whereas visual-only or language-only masking scores well on the bias metric only because the model names a gender in under 10% of cases.
• mechanismsingle paper↑1↓0
  When a multimodal model is asked an object-existence question over a set of images at once, it is less accurate than when the same images are queried one at a time and the answers combined, so cross-image integration adds errors beyond the per-image perception errors.
• observationsingle paper↑1↓0
  In open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.
• observationsingle paper↑1↓0
  In open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.
• mechanismsingle paper↑1↓0
  For panoramic dental X-ray VQA with frozen vision-language models, replacing the model's own spatial judgement with external detector findings that are deterministically bound to tooth identities before synthesis raises answer accuracy over the same two-pass pipeline without detectors, and the resulting accuracy is bounded by detector quality — degrading the identity/attribution detector costs more downstream accuracy than a much larger degradation of a task-specific pathology detector.
• mechanismsingle paper↑1↓0
  In region-guided contrastive decoding for medical VLMs, the effectiveness of the guidance depends on how much of the image the guiding mask covers: coarse ground-truth boxes (mean ~39% coverage on SLAKE) suppress so much of the unconditional branch that the contrast weakens, while compact automatically discovered masks (~7% coverage) selected by the drop in answer likelihood under occlusion give a sharper signal and higher closed-ended accuracy.
• mechanismsingle paper↑1↓0
  For short-answer object-existence questions in open multimodal models (LLaVA v1.5, MiniGPT-4, Qwen2.5-VL), object hallucination is driven partly by the visual encoder itself rather than only by language priors: hallucinated cases show lower image-text embedding similarity and inverted attention, and swapping in a weaker or noisier visual encoder lowers POPE accuracy while a stronger one raises it.
• mechanismsingle paper↑1↓0
  For short-answer object-existence questions in open multimodal models (LLaVA v1.5, MiniGPT-4, Qwen2.5-VL), object hallucination is driven partly by the visual encoder itself rather than only by language priors: hallucinated cases show lower image-text embedding similarity and inverted attention, and swapping in a weaker or noisier visual encoder lowers POPE accuracy while a stronger one raises it.
• mechanismsingle paper↑1↓0
  In training-free contrastive decoding for 7B-class medical VLMs, the spatial and temporal contrast terms act on different answer types: anatomy-conditioned classifier-free guidance is what raises closed-ended accuracy (68.5 to 81.5 on SLAKE with Phi-3.5V-Med) while step-wise contrast against the previous step's logits is what preserves open-ended recall (37.8 to 44.9), so dropping either term costs about five overall points.
• observationsingle paper↑1↓0
  For training-free inference-time hallucination interventions in 7B-class vision-language models, increasing intervention strength (finer regional evidence partitioning, longer attention-routing windows, stronger textual perturbation) keeps lowering CHAIR object-hallucination scores while producing empty or reduced-coverage responses, so lower hallucination numbers at high intervention strength partly reflect the model saying less rather than grounding better.

Techniques

• Adversarial contrastive fine-tuning on hallucination-flippedprocess
  Adversarial contrastive fine-tuning on hallucination-flipped image pairs, as introduced by the paper it was drafted from.
• Attenuate hidden-state modes by hallucination-to-signal ratiarchitecture
  Estimate truthful and hallucination-distortion covariances from paired captions offline, build a closed-form Wiener gain per eigenmode of the distortion covariance, and multiply it into the FFN output projection of deep layers so inference
• Calibrate value vectors in synergy attention headsdecoding
  Classify attention heads by causal noise intervention and counterfactual difference-in-differences, then at decode time rescale the visual and language value vectors of the mixed 'synergy' heads so their information ratio hits a target equi
• Check logit-lens decode of attended image patches against thdecoding
  At each generated object token, average attention over mid-to-late layers, decode the top-k attended image tokens through the output head, and flag hallucination when no decoded token is semantically similar to the emitted word; then either
• Detect hallucination from adjacent-layer attention drift plutooling
  Compute KL divergence between consecutive layers' cross-modal attention over visual tokens, mask the top-attended region of the layer just before the largest drift, re-query the same token's probability, and feed the drift profile plus the
• Diagnosis-selected head-restricted adapter plus grounding-gaprocess
  Diagnosis-selected head-restricted adapter plus grounding-gated decoding, as introduced by the paper it was drafted from.
• Fuse CLIP object logits by LVLM entropy at decodingdecoding
  At each decoding step, pick object tokens among the top-k candidates with WordNet, rescore them with CLIP similarity to the image, rescale the scores into the LVLM logit range, and blend the two with a weight equal to the LVLM's normalized
• Hidden-state divergence compensation at decoding timeprocess
  Hidden-state divergence compensation at decoding time, as introduced by the paper it was drafted from.
• Mask language-only self-attention, keep vision-to-language fdecoding
  At inference, apply mode-specific attention masks that suppress language-only self-attention in the late decoder layers so text tokens are forced to draw on visual tokens when attributing person attributes.
• Penalize tokens whose pooled rank evidence falls below theirdecoding
  At prefill, pool each candidate token's reciprocal vocabulary rank across all visual-token logit-lens readouts, cache it, and at each decoding step apply a bounded logit penalty to candidates whose pooled rank evidence falls below a token-s
• Penalize tokens with a low vision-ablation log-prob gapdecoding
  Run a second decoding pass with the visual features zeroed, score each candidate token by the log-probability difference between the two paths, and subtract a decaying penalty from candidates whose score falls below a threshold.
• Pick the contrast region by counterfactual occlusion, penalidecoding
  Propose anatomical masks with a medical segmentation model, score each by the log-likelihood drop of a draft answer under two occlusion operators minus an area penalty, and use the top compact mask for classifier-free-guidance plus step-wis
• Read the whole computational trace as a graph, not one hiddearchitecture
  Builds a directed attention graph over image patches, query tokens and generated tokens from a single forward pass of a frozen LVLM and classifies each response token with alternating GNN, ViT and GRU blocks, with a streaming variant that r
• Reward calibration of stated confidence inside GRPOtraining
  Augments the output schema with span and answer confidence values and trains with GRPO whose reward adds a Brier penalty against realised rollout correctness, a group-level ECE bonus, an abstention term supervised by group accuracy, and a s
• Verify object claims by cross-query semantic and spatial agrtooling
  For each object claim, query a frozen open-vocabulary detector with four paraphrased prompts, average the top scores for semantic support, score whether the queries' preferred boxes form one overlapping cluster with a dominant candidate, fu
• Weight several hallucination subspaces per image, not one gldecoding
  Offline, cluster hallucinated-vs-faithful hidden-state difference vectors and take an SVD basis per cluster; at test time probe the model with the image and a 70%-masked copy, score each subspace by the projection of the hidden-state differ

Sources

Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.

• Semantic-Spatial Agreement Verification for Mitigating Object Hallucination in Multimodal Large Language Models2026-09-15
  ↑ supports a claim here
• What Do Hallucinations Reveal About Multimodal Reasoning? Diagnosing Visual Grounding Failures via Contrastive Decoding Probes2026-09-15
  ↑ supports a claim here
• SAVOR: Self-Aware Visual Grounding via Confidence-Calibrated Reinforcement Learning for Multimodal Hallucination Mitigation2026-09-15
  ↑ supports a claim here
• ViD: Vision-Dominant Gender Bias Mitigation for Large Vision-Language Models2026-09-15
  ↑ supports a claim here
• AgenTeeth: A Model-Agnostic Framework for Suppressing Hallucination in Frozen Vision-Language Models on Dental X-Rays via Tool Evidence Injection2026-09-15
  ↑ supports a claim here
• MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads2026-09-05
  ↑ supports a claim here
• Detecting Object Hallucinations in Large Vision-Language Models via Cross-Modal Attention Drifts and Mask-Based Verification2026-09-02
  ↑ supports a claim here
• RVSD: Retrieval Vision Sparse Decoding for Mitigating Visual Hallucinations in Large Vision-Language Models2026-09-02
  ↑ supports a claim here
• Reliability Challenges in Diffusion Vision-Language Models2026-09-01
  ↑ supports a claim here
• VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMs2026-08-31
  ↑ supports a claim here
• Fine-Grained Multi Image Object Hallucination Benchmark2026-08-31
  ↑ supports a claim here
• Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM2026-08-31
  ↑ supports a claim here · ↑ supports a claim here
• Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilization2026-08-30
  ↑ supports a claim here
• EviAnchor: Mitigating Hallucinations in Large Vision-Language Models via Regional Visual Evidence Compensation2026-08-29
  ↑ supports a claim here
• Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Models2026-08-28
  ↑ supports a claim here · ↑ supports a claim here
• ReVA: A Region-Aware Visual Assistant for Visually Grounded Question Answering2026-08-27
  about this capability · no claim drafted yet
• Targeting the Attention Heads Behind Object Hallucination in LLaVA2026-08-25
  ↑ supports a claim here · ↑ supports a claim here
• PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment2026-08-20
  ↑ supports a claim here
• ReWEIGH the Evidence: Calibrating Token-Level Ordinal Visual Evidence to Mitigate Hallucinations in Large Vision-Language Models2026-08-19
  ↑ supports a claim here · ↑ supports a claim here
• Counterfactual Anatomy-guided Spatial-Temporal Decoding for Annotation-Free Hallucination Mitigation in Medical VLMs2026-08-18
  ↑ supports a claim here · ↑ supports a claim here
• Test-Time Hallucination Control in Large Vision-Language Models2026-08-11
  ↑ supports a claim here
• UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations2026-08-11
  ↑ supports a claim here
• Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs2026-08-10
  ↑ supports a claim here
• Wiener Representation Filtering for VLM Hallucination Suppression2026-08-08
  ↑ supports a claim here
• Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination2026-08-07
  ↑ supports a claim here
• TruthLens: Object Hallucination Detection via Self-Evaluating Truthfulness Scores in LVLMs2026-08-06
  about this capability · no claim drafted yet