open-ended-emotion-interpretation-multimodal-llms-coarse-grained-hallucination-mitigations-designed CLM-0460For open-ended emotion interpretation by multimodal LLMs, coarse-grained hallucination mitigations designed for object hallucination (VCD, ICD, M3ID) and self-verification (EmotionHallucer) cut unsupported claims in some facets while raising them in others — e.g. lowering audio hallucination but increasing expression and action hallucination — so a single aggregate hallucination rate hides the trade-off.
Capability: Stating false facts confidently
Observed on
Free-form emotion reasoning over video/audio/text on OV-MERD+, CMU-MOSI and CH-SIMS, with AffectGPT, Qwen2.5-Omni and R1-Omni backbones, scored by an LLM-based claim atomizer plus .
Sources
- Validated by: Measured facet-wise and total Emotion Hallucination Rate against VCD, ICD, M3ID, EmotionHallucer and Best-of-N on three backbones and three datasets: total rate down 3.27-11.85 points, the only method reducing all six facets everywhere, with F1/recall on emotion prediction also rising; the evaluator itself agreed with three human annotators on 100 samples (Pearson 0.84, Spearman 0.92).. Measured on 532 OV-MERD+ samples plus CMU-MOSI and CH-SIMS, three backbones, four baselines. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In 7-8B multimodal models, object hallucination during visually grounded token generation tracks a shift of the visual-language information ratio toward language inside mixed-modality "synergy" attention heads rather than a shortage of visual-specific heads, and rescaling visual versus language value vectors in those heads causally moves hallucination in both directions.Grounding answers in the image · reviewed by AI
- In 7B-class LVLMs (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2), hallucination directions in activation space are not a single mode: clustering hallucinated-minus-faithful hidden-state differences into several low-rank subspaces and weighting them per test image lowers object-hallucination rates slightly below a single global null-space projection, with the margin small (CHAIR_I 5.30 to 4.92 on LLaVA-1.5) and inconsistent on yes/no existence probing.Grounding answers in the image · reviewed by AI
- For training-free inference-time hallucination mitigation in 7B vision- language models, reported gains on object-hallucination benchmarks are largely inseparable from reduced informativeness: hallucination rate and object recall/coverage move together, so a lower score can mean the model mentioned fewer visual entities rather than grounded better.Whether the measurement made the finding · reviewed by AI
- In open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.Grounding answers in the image · reviewed by AI
- For encoder-based hallucination/claim-faithfulness detectors moved from general-domain data to biomedical claim verification, which corpus the encoder was pre-trained on matters more than which data it is fine-tuned on: a PubMed-pretrained encoder fine-tuned on a few hundred SciFact pairs beat both a general encoder fine-tuned on the same pairs and a general encoder pre-fine-tuned on a large general hallucination set.Competence in a specialist domain · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Rerunning the same mitigation baselines on these backbones and finding they reduce every facet-wise hallucination rate, or showing the facet increases come from the LLM judge's labelling rather than the generations.