long-video-10-150-minute-audio-visual-reasoning-questions-multiple-choice-accuracy-overstates CLM-0434observationsingle paperReviewed by AI
On long-video (10-150 minute) audio-visual reasoning questions, multiple-choice accuracy overstates grounding: a large open model can answer about 55% of items correctly while scoring 8.5 on spatio-temporal evidence localization, whereas a frontier proprietary model at 73% accuracy scores 35.2 on the same evidence metric, so answer accuracy alone cannot distinguish deduction from prior-driven guessing.
Capability: Reasoning about time in video
Observed on
Long cinematic videos requiring both audio and visual cues; evidence scored by soft temporal alignment plus LLM-judged semantic matching against annotated timestamps..
Sources
- Validated by: Applied to three models on 4,000 QA pairs over 784 videos; separated a 55.1%-accuracy open model (evidence 8.5) from a 73.1%-accuracy proprietary model (evidence 35.2). No human validation of the evidence metric reported.. Measured on the paper's own benchmark with an LLM-judged semantic component and hand-set weights; only three models reported for the evidence comparison, smaller models excluded for near-zero evidence scores, so the gap could partly reflect output-format compliance rather than grounding.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On month-level egocentric recordings (20-120 days per participant) with human-written multiple-choice questions, current multimodal models degrade as the question moves from inferring recurring habits, to retrieving one specific episode, to composing evidence across days — the best model reached 71.8% macro-average against a 94.2% human baseline, with several models near or below the 25% chance level on route reasoning, cross-view spatial reasoning and direction judgement.Reasoning about time in video · reviewed by AI
- For large audio language models on multiple-choice audio reasoning benchmarks (MMAR, MMAU-mini), accuracy does not track whether the model's chain of thought is acoustically grounded or causally drives the answer: several models answer correctly from perception steps that assert unsupported labels, and for some models replacing the reasoning chain with vacuous or fabricated content leaves the final answer unchanged.Understanding speech and audio · reviewed by AI
- For spoken language models answering questions over long dialogue histories, converting the history to transcripts beats keeping it as audio when the answer is a lexical detail (exact slot values across dense turns: mean 71.3% vs 42.6%), but loses when the answer depends on how something was said, where audio history wins for six of seven models and by a larger margin once lexical emotion cues are removed.Understanding speech and audio · reviewed by AI
- On adversarial video hallucination pairs scored only when a method both accepts the factual statement and rejects the counterfactual one, purpose-built video hallucination detectors (embedding-similarity, learned-entailment, atomic-fact and attention-based) top out around 34.63% overall accuracy, well below a general frontier video-language model prompted to judge the same items (about 83.63%).Stating false facts confidently · reviewed by AI
- When Video-LLMs are asked to verify sentence by sentence whether a dense video caption is supported by the video, they detect hallucinated sentences less often in captions they generated themselves than in captions from other models, and detection of temporal errors (action, order, causality, frequency, state change) improves much less with model scale than detection of static-frame errors.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Models' evidence-localization scores tracking their MCQ accuracy closely (high rank correlation, similar relative gaps), or human raters judging the low-evidence model's cited timestamps as valid.