long-video-qa-under-fixed-32-frame-budget-7b-8b-multimodal-backbones CLM-0605observationsingle paperReviewed by AI
For long-video QA under a fixed 32-frame budget with 7B-8B multimodal backbones, selecting frames from the structural shape of the frame-query similarity curve (peak, rising, falling, transition regions) beats ranking frames by similarity magnitude, and the margin grows with video length — a few points on short clips and roughly 8 points on hour-long videos — while on short/medium-clip benchmarks it is near-indistinguishable from other selectors.
Capability: Reasoning about time in video
Observed on
Training-free post-processing over precomputed frame-query scores; four long-video QA benchmarks, three backbones (Qwen2.5-VL-7B, InternVL-3-8B, LLaVA-OV-7B), budget 8-64 frames..
Sources
- Validated by: Accuracy on Video-MME, LongVideoBench, MLVU, LVBench against uniform sampling and AKS, BOLT, MDP3, FOCUS; best in most of 12 model-benchmark cells; matched-scorer controls (CLIP, SigLIP) show +3.5 and +3.8 average points over BOLT and MDP3; component ablations each cost 0.6-1.6 points.. Measured on four benchmarks, three backbones, budgets 8-64, four LVLM scales, five scorers. Main table is not shared-scorer; two matched-scorer blocks (8 cells, RIDGE higher in 7) partly isolate the policy. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For long-video QA under a fixed frame budget with a frozen 7B multimodal model, organizing frames into a question-agnostic hierarchical temporal tree and searching it coarse-to-fine (answer, then issue a search statement, then expand the relevant branch) raises accuracy over uniform sampling and over agent methods that score frames independently, with the gain largest on the longest-video benchmarks.Reasoning about time in video · reviewed by AI
- For long-video QA with a fixed frame budget fed to a frozen multimodal model, selecting frames by marginal contribution to the already-selected subset (query relevance plus coverage gain minus redundancy) beats ranking frames independently by query-frame similarity, and the margin over uniform sampling grows with video duration.Reasoning about time in video · reviewed by AI
- For 7-8B multimodal models answering multiple-choice long-video questions under a fixed 16-frame budget, selecting keyframes by embedding similarity to the query concatenated with each candidate answer beats query-only scoring, query-agnostic diversity selection, and uniform sampling across Qwen2-VL, LLaVA-Video and InternVL3.5, with the largest margins on the longest video splits — but per-category results still invert, including uniform sampling winning some categories.Reasoning about time in video · reviewed by AI
- In training-free video temporal grounding with frozen vision-language embedding backbones, matching a query only against frames sampled at a fixed rate leaves temporal context unused: adding clip-level (sliding-window) similarities projected onto the frame timeline raises localization accuracy even though clip-only similarity is worse than frame-only on its own.Reasoning about time in video · reviewed by AI
- For streaming video understanding with 7-8B multimodal backbones, question-relevant historical frames can be retrieved from the keys of only the first few language-Transformer layers (layer ~3-4 of 28-32), so continuous frames need not be prefilled through full model depth: indexing shallow and re-prefilling only the retrieved units at full depth cuts per-frame prefill up to 52x and 20-second end-to-end latency up to 15x while scoring about 1.7 points above the best same-backbone baseline on OVO-Bench and StreamingBench.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Under matched scorers and budgets, curve-shape region selection fails to beat magnitude ranking on long-horizon benchmarks, or its advantage does not increase with video duration.