adversarial-video-hallucination-pairs-scored-only-method-both-accepts CLM-0356observationsingle paperReviewed by AI
On adversarial video hallucination pairs scored only when a method both accepts the factual statement and rejects the counterfactual one, purpose-built video hallucination detectors (embedding-similarity, learned-entailment, atomic-fact and attention-based) top out around 34.63% overall accuracy, well below a general frontier video-language model prompted to judge the same items (about 83.63%).
Capability: Stating false facts confidently
Observed on
2,000 adversarial samples over 1,090 real-world videos, eight hallucination categories across video QA and captioning, fifteen methods evaluated under one protocol; overall accurac.
Sources
- Validated by: Applied to fifteen detection methods on 2,000 paired samples; separates factual, counterfactual and joint accuracy, showing dedicated detectors peak at 34.63% overall. No comparison against an independent human-detector reference for the protocol itself.. Measured on a benchmark the authors built with an LLM multi-agent pipeline; category labels audited at 98.75% sample accuracy on a subset. The result table is truncated in the source, so only the headline peak figures (34.63% dedicated, 83.63% Gemini-3-Flash) are available. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For training-free inference-time hallucination mitigation in 7B vision- language models, reported gains on object-hallucination benchmarks are largely inseparable from reduced informativeness: hallucination rate and object recall/coverage move together, so a lower score can mean the model mentioned fewer visual entities rather than grounded better.Whether the measurement made the finding · reviewed by AI
- In synthetic hallucination benchmarks where faithful items are human-written and hallucinated items are LLM-rewrites, a detector's binary score partly measures human-vs-machine style rather than factuality: feeding faithful LLM rewrites of the same truthful articles makes some fine-tuned encoders flag nearly all of them as hallucinated.Whether the measurement made the finding · reviewed by AI
- When Video-LLMs are asked to verify sentence by sentence whether a dense video caption is supported by the video, they detect hallucinated sentences less often in captions they generated themselves than in captions from other models, and detection of temporal errors (action, order, causality, frequency, state change) improves much less with model scale than detection of static-frame errors.Reasoning about time in video · reviewed by AI
- For VLM-based judging of camera-conditioned world-model videos, splitting a long video pair into action-aligned chunks with structured visual evidence (source image, frame-grid overview, per-action first/last-frame panels) and voting chunk verdicts into video-level preferences agrees with human preferences better than judging the whole video pair at once, because short-lived action evidence is otherwise diluted by the full context.Judging another's work as well as a careful reviewer would · reviewed by AI
- On long-video (10-150 minute) audio-visual reasoning questions, multiple-choice accuracy overstates grounding: a large open model can answer about 55% of items correctly while scoring 8.5 on spatio-temporal evidence localization, whereas a frontier proprietary model at 73% accuracy scores 35.2 on the same evidence metric, so answer accuracy alone cannot distinguish deduction from prior-driven guessing.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A dedicated video hallucination detector scoring near or above general-purpose VLM judges on joint factual/counterfactual accuracy on this or an equivalent paired benchmark.