multi-hop-video-questions-written-no-single-frame-or-timestamp CLM-0449On multi-hop video questions written so no single frame or timestamp lookup suffices, single-turn MLLMs answer poorly (0.07-0.51 accuracy across 12 proprietary and open models), and wrapping the same models in an agentic loop helps only when the loop lets the agent choose what, where and at what granularity to look: frameworks giving adaptive fps/resolution/interval control raised accuracy and trajectory scores for nearly every model, while a fixed-schedule framework with weak perception tools (STAR) fell below the single-turn baseline for all of them.
Capability: Reasoning about time in video
Observed on
100 hand-authored 26-option multiple-choice questions over 71 openly licensed videos of 8s-27min; accuracy plus an LLM-judged milestone trajectory score; only four agentic framewor.
Sources
- Validated by: Accuracy and trajectory score on AgentVidBench against single-turn and three agentic frameworks; best accuracy 0.58 with Gemini 2.5 Pro versus 0.51 single-turn and 0.56 AVP, with ablations removing each control axis lowering accuracy (e.g. 0.50 without interval control).. Measured across 12 models x 5 settings on one 100-question benchmark built by the same authors; the STAR shortfall confounds framework design with the quality of that framework's bundled perception tools, and trajectory scores come from an LLM judge validated only in an appendix.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For 7B multimodal models on long-video QA, a multi-turn agentic exploration harness (adaptive viewing speed, cross-turn textual notes, rewatch) lowers accuracy below plain uniform-frame single-pass answering unless the model is trained to use it, because the untrained model commits to an answer before gathering evidence; with cold-start SFT on verified exploration turns plus trajectory-level RL, the same harness beats the single-pass baseline.Reasoning about time in video · reviewed by AI
- For long-video QA under a fixed frame budget with a frozen 7B multimodal model, organizing frames into a question-agnostic hierarchical temporal tree and searching it coarse-to-fine (answer, then issue a search statement, then expand the relevant branch) raises accuracy over uniform sampling and over agent methods that score frames independently, with the gain largest on the longest-video benchmarks.Reasoning about time in video · reviewed by AI
- For 4-8B multimodal models on long-video QA, RL training on questions authored from short scene segments but answered over the whole video improves benchmark accuracy more than RL on questions authored from the whole video, because segment-authored questions target details a whole-video view overlooks; including the segment timestamp in the training instruction adds a further gain by removing referential ambiguity, and is not needed at inference.Reasoning about time in video · reviewed by AI
- For deepfake video detection with 7-8B open multimodal backbones, splitting the analysis into four separately prompted perspective agents (texture, lighting, motion, physics) whose reports a judge agent reconciles beats the same backbone doing chain-of-thought or multi-turn self-questioning over the same four perspectives, and the gain is not explained by extra turns or repeated frame access.Getting more from several models than from one · reviewed by AI
- For multi-task procedural video understanding with a small (1B) video-LLM decoder, replacing each decoder feed-forward block with task-specific expert copies and routing each video-prompt sample by its known task identity gives higher average top-1 accuracy across the five COIN tasks than the same expert topology run densely or with learned token-level or sample-level routing, with the largest gains on the procedure-forecasting tasks.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A replication where fixed-plan agentic frameworks match adaptive-control ones on these questions, or where single-turn MLLMs match the best agentic setups on multi-hop video QA. Drafted stance toward 7b-multimodal-models-long-video-qa-multi-turn-agentic-exploration-harness: supports -- Both find that an agentic video-exploration harness can score below plain single-pass answering when the loop or its tools are poorly matched to the model.