video-llms-asked-verify-sentence-sentence-whether-dense-video-caption CLM-0504When Video-LLMs are asked to verify sentence by sentence whether a dense video caption is supported by the video, they detect hallucinated sentences less often in captions they generated themselves than in captions from other models, and detection of temporal errors (action, order, causality, frequency, state change) improves much less with model scale than detection of static-frame errors.
Capability: Reasoning about time in video
Observed on
500 videos, 4s to 90min, five spatio-temporal complexity types; captions generated by five Video-LLMs with human sentence-level labels; open models 2B-72B plus Gemini 3 Flash and G.
Sources
- Measured: per-sentence precision/recall/F1 averaged over video-caption pairs, cross-source vs self-source heatmap, and recall-gain heatmap across three model families. Best model (Gemini 3 Flash) F1 above 40 on other models' captions and roughly half that on its own. All models score far below ceiling (best F1 ~42), so the self-preference gap is measured in a low-absolute-performance regime. No control for caption style differences between generators.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When vision-language models are given an explicit abstention option plus a stated payoff for correct, wrong and declined answers, quadrupling the penalty for a wrong answer barely changes how often they answer, so their abstention rate is largely insensitive to the stated cost of error across text, synthetic video and natural video recall tasks.Saying it does not know · reviewed by AI
- On adversarial video hallucination pairs scored only when a method both accepts the factual statement and rejects the counterfactual one, purpose-built video hallucination detectors (embedding-similarity, learned-entailment, atomic-fact and attention-based) top out around 34.63% overall accuracy, well below a general frontier video-language model prompted to judge the same items (about 83.63%).Stating false facts confidently · reviewed by AI
- On long-video (10-150 minute) audio-visual reasoning questions, multiple-choice accuracy overstates grounding: a large open model can answer about 55% of items correctly while scoring 8.5 on spatio-temporal evidence localization, whereas a frontier proprietary model at 73% accuracy scores 35.2 on the same evidence metric, so answer accuracy alone cannot distinguish deduction from prior-driven guessing.Reasoning about time in video · reviewed by AI
- In 7B-class vision-language models (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2), object hallucination is carried by an anisotropic set of directions in deep-layer hidden states, so attenuating each eigenmode in proportion to its hallucination-to-truthful variance ratio — estimated once offline from paired truthful/hallucinated captions and folded into the FFN output projection — lowers CHAIR rates, while a mean-offset subtraction or isotropic shrinkage of the same representations does not.Grounding answers in the image · reviewed by AI
- When an LLM judge is given a prior evaluation score as context metadata on a revision, its rubric scores shift toward that uninformative prior — seven of eight tested models (API and open-weight, 0-5 scale, 20 fixed texts) show a negative total shift, and on a categorical compliance task the metadata blocks 48% of error corrections and flips about 10% of previously correct judgments to the anchored wrong label.Judging another's work as well as a careful reviewer would · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A Video-LLM that scores the same or better F1 on verifying its own captions as on other models' captions on the same videos, or a model family where scaling raises recall on temporal error categories as much as on static ones.