7-8b-multimodal-models-answering-multiple-choice-long-video-questions-under-fixed CLM-0193For 7-8B multimodal models answering multiple-choice long-video questions under a fixed 16-frame budget, selecting keyframes by embedding similarity to the query concatenated with each candidate answer beats query-only scoring, query-agnostic diversity selection, and uniform sampling across Qwen2-VL, LLaVA-Video and InternVL3.5, with the largest margins on the longest video splits — but per-category results still invert, including uniform sampling winning some categories.
Capability: Reasoning about time in video
Observed on
Training-free, MLLM-agnostic frame-selection modules; three 7-8B open MLLMs; MLVU, Video-MME, LongVideoBench; 16 selected frames (4+8+32 multi-resolution for Q-Frame); CLIP/LongCLI.
Sources
- supportsEvaluation of MLLM-Agnostic Plug-and-Play Keyframe Selection Methods for Long Video UnderstandingHead-to-head re-run of five published selection methods with released code on 3 MLLMs x 3 benchmarks (15 aggregate settings, 546 table entries); the answer-aware method is best in 13 and second in 2, with reported gains over uniform sampling of 8.8/7.8/5.11 points on MLVU. Single-lab reproduction, no seeds or variance reported, and the winning method is a technical report by an overlapping line of work; per-task tables show category-level reversals, so the aggregate ranking is not uniform across skills.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For long-video QA with a fixed frame budget fed to a frozen multimodal model, selecting frames by marginal contribution to the already-selected subset (query relevance plus coverage gain minus redundancy) beats ranking frames independently by query-frame similarity, and the margin over uniform sampling grows with video duration.Reasoning about time in video · reviewed by AI
- For 4-8B multimodal models on long-video QA, RL training on questions authored from short scene segments but answered over the whole video improves benchmark accuracy more than RL on questions authored from the whole video, because segment-authored questions target details a whole-video view overlooks; including the segment timestamp in the training instruction adds a further gain by removing referential ambiguity, and is not needed at inference.Reasoning about time in video · reviewed by AI
- For long-video QA under a fixed frame budget with a frozen 7B multimodal model, organizing frames into a question-agnostic hierarchical temporal tree and searching it coarse-to-fine (answer, then issue a search statement, then expand the relevant branch) raises accuracy over uniform sampling and over agent methods that score frames independently, with the gain largest on the longest-video benchmarks.Reasoning about time in video · reviewed by AI
- For long-video QA under a fixed 32-frame budget with 7B-8B multimodal backbones, selecting frames from the structural shape of the frame-query similarity curve (peak, rising, falling, transition regions) beats ranking frames by similarity magnitude, and the margin grows with video length — a few points on short clips and roughly 8 points on hour-long videos — while on short/medium-clip benchmarks it is near-indistinguishable from other selectors.Reasoning about time in video · reviewed by AI
- For 7B multimodal models on long-video QA, a multi-turn agentic exploration harness (adaptive viewing speed, cross-turn textual notes, rewatch) lowers accuracy below plain uniform-frame single-pass answering unless the model is trained to use it, because the untrained model commits to an answer before gathering evidence; with cold-start SFT on verified exploration turns plus trajectory-level RL, the same harness beats the single-pass baseline.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A head-to-head run under the same fixed frame budget where answer-conditioned frame scoring does not beat query-only scoring or uniform sampling on aggregate benchmark accuracy, or where its advantage disappears on the longest-duration splits. Proposed technique, not catalogued: answer-aware keyframe selection.