diffusion-vision-language-models-generating-free-form-captions-object-tokens-hallucinatedIn diffusion vision-language models generating free-form captions, object tokens that are hallucinated commit at later denoising steps and with lower per-token confidence than grounded object tokens, so the commit-step trajectory is an available internal hallucination signal that has no counterpart in autoregressive decoding.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Grounding answers in the image
Observed on
Two LaViDa backbones (LLaDA and Dream), 500 MSCOCO val2014 captions, 128 denoising steps, hallucination labelled by CHAIR object annotations..
Sources
- Measured: mean commit step 63.8 vs 37.6 (LaViDa-L) and 64.3 vs 42.2 (LaViDa-D) on a 128-step budget; commit-step ROC-AUC 0.699 and 0.667, PR-AUC 0.374 and 0.261 against base rates 0.190 and 0.152, 5-fold cross-validated. Only two backbones, one dataset, one caption prompt. An attention-based analysis found no difference, so the mechanism behind the signal is not established.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- In 7B vision-language models that prune visual tokens during decoding for efficiency, hallucination rises as the generated caption gets longer, because pruning is irreversible and discards fine-grained evidence needed at later decoding steps; keeping the pruned tokens as a retrievable bank and pulling them back when next-token entropy spikes recovers the grounding without a second forward pass.Grounding answers in the image · unreviewed
- In open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.Grounding answers in the image · unreviewed
- For black-box hallucination detection using only sampled responses and token log-probabilities, semantic-entropy signals go to zero whenever the samples collapse into one semantic cluster — which happens for a large share of hallucinated queries, up to nearly all of them on tasks with confusable entities and figures — while token-level uncertainty still separates many of those cases, so the two signals fail on different queries and combining them detects more than either alone.Stating false facts confidently · unreviewed
- For training-free inference-time hallucination mitigation in 7B vision- language models, reported gains on object-hallucination benchmarks are largely inseparable from reduced informativeness: hallucination rate and object recall/coverage move together, so a lower score can mean the model mentioned fewer visual entities rather than grounded better.Whether the measurement made the finding · unreviewed
- For detecting hallucinated text spans in vision-language model outputs, span localization and confidence calibration come from different architectures: a fine-tuned generative VLM proposes better span boundaries but scores them poorly, while a discriminative token tagger trained with a direct regression loss on human-consensus confidence gives better-correlated probabilities but conservative spans, so using the generative spans as proposals and the tagger's probabilities as scores beats either alone.Stating false facts confidently · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: On other diffusion VLMs or datasets, commit step for hallucinated object tokens is no later than for grounded ones, or ROC-AUC using commit step falls to chance. Proposed technique, not catalogued: Commit-step and confidence trajectory as a hallucination detector.