streaming-video-understanding-7-8b-multimodal-backbones-question-relevant-historical-frames CLM-0644For streaming video understanding with 7-8B multimodal backbones, question-relevant historical frames can be retrieved from the keys of only the first few language-Transformer layers (layer ~3-4 of 28-32), so continuous frames need not be prefilled through full model depth: indexing shallow and re-prefilling only the retrieved units at full depth cuts per-frame prefill up to 52x and 20-second end-to-end latency up to 15x while scoring about 1.7 points above the best same-backbone baseline on OVO-Bench and StreamingBench.
Capability: Reasoning about time in video
Observed on
Qwen3-VL-8B and LLaVA-OneVision-7B, 1 FPS streaming, OVO-Bench/StreamingBench/LVBench, no task-specific training; retrieval gate threshold calibrated once off-benchmark..
Sources
- Validated by: OVO-Bench and StreamingBench accuracy against online MLLMs and training-free offline-to-online baselines (ReKV, HERMES, SimpleStream, OASIS, CausalMem, LiveVLM) on two backbones, plus measured per-frame prefill, 20s end-to-end latency and peak GPU memory on an RTX 5090; ablations isolate the gate (48.7% retrieval on Backward vs 4.9% on Real-Time), token voting (59.48% vs 58.91% pooled Q-K, 57.47% SigLIP) and diversity (59.77% to 61.21%).. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In training-free video temporal grounding with frozen vision-language embedding backbones, matching a query only against frames sampled at a fixed rate leaves temporal context unused: adding clip-level (sliding-window) similarities projected onto the frame timeline raises localization accuracy even though clip-only similarity is worse than frame-only on its own.Reasoning about time in video · reviewed by AI
- On hour-scale (mean 88.8 min) open-ended streaming video QA with causal access, every streaming memory design degrades as the evidence-to-query gap grows, and they fail in different ways: recent-window systems cannot recover distant events (AURA 25.4% within 5 minutes falling to 10.5% beyond 30 minutes), text-summary systems keep temporal coverage but discard visual evidence, and model-internal compression loses fine-grained detail (StreamForest 14.4%, ThinkStream 7.5% on retrospection), while human accuracy without rewatching is 63.4%.Reasoning about time in video · reviewed by AI
- For long-video QA under a fixed frame budget with a frozen 7B multimodal model, organizing frames into a question-agnostic hierarchical temporal tree and searching it coarse-to-fine (answer, then issue a search statement, then expand the relevant branch) raises accuracy over uniform sampling and over agent methods that score frames independently, with the gain largest on the longest-video benchmarks.Reasoning about time in video · reviewed by AI
- For long-video QA under a fixed 32-frame budget with 7B-8B multimodal backbones, selecting frames from the structural shape of the frame-query similarity curve (peak, rising, falling, transition regions) beats ranking frames by similarity magnitude, and the margin grows with video length — a few points on short clips and roughly 8 points on hour-long videos — while on short/medium-clip benchmarks it is near-indistinguishable from other selectors.Reasoning about time in video · reviewed by AI
- For 4-8B multimodal models on long-video QA, RL training on questions authored from short scene segments but answered over the whole video improves benchmark accuracy more than RL on questions authored from the whole video, because segment-authored questions target details a whole-video view overlooks; including the segment timestamp in the training instruction adds a further gain by removing referential ambiguity, and is not needed at inference.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Show that retrieval quality from shallow-layer keys falls materially below full-depth retrieval on streams or backbones where the layer-wise diagnostic was not run, or that the latency gains disappear once archive updates, transfer and re-prefill are counted, or that accuracy drops below strong recent-window baselines on backward-tracing categories.