streaming-video-question-answering-frozen-video-llm-appending-retrieved-historical CLM-0159For streaming video question answering with a frozen video-LLM, appending retrieved historical frames as extra visual context helps less than optimizing a small set of latent memory tokens at test time and then dropping the retrieved tokens before decoding; the gain is largest on questions requiring backward tracing over past history.
Capability: Reasoning about time in video
Observed on
Frozen Qwen2.5-VL 3B/7B backbones, streaming and long-video benchmarks with a bounded memory budget; test-time optimization only, no parameter updates..
Sources
- Ablation on OVO-Bench/VideoMME: retrieved-evidence-appended 59.7/65.6 vs evolved latent tokens 64.2/66.6, unoptimized latent tokens 56.5/65.2, baseline 56.9/65.4. Backward Tracing rose 44.7 to 60.0 while real-time perception rose 63.3 to 68.5. Single paper, authors' own system, two backbones; does not isolate which of the three components carries the effect beyond the reported staged ablation.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On hour-scale (mean 88.8 min) open-ended streaming video QA with causal access, every streaming memory design degrades as the evidence-to-query gap grows, and they fail in different ways: recent-window systems cannot recover distant events (AURA 25.4% within 5 minutes falling to 10.5% beyond 30 minutes), text-summary systems keep temporal coverage but discard visual evidence, and model-internal compression loses fine-grained detail (StreamForest 14.4%, ThinkStream 7.5% on retrospection), while human accuracy without rewatching is 63.4%.Reasoning about time in video · reviewed by AI
- For long cooking videos where a question depends on ingredient changes no longer visible, vision-language models fail to carry observed events forward into later answers, and adding generic retrieved-context RAG makes it worse, while splitting memory into a temporally anchored event log plus an explicitly updated inventory ledger recovers much of the gap.Tracking state through a long task · reviewed by AI
- For 4-8B multimodal models on long-video QA, RL training on questions authored from short scene segments but answered over the whole video improves benchmark accuracy more than RL on questions authored from the whole video, because segment-authored questions target details a whole-video view overlooks; including the segment timestamp in the training instruction adds a further gain by removing referential ambiguity, and is not needed at inference.Reasoning about time in video · reviewed by AI
- For streaming video understanding with 7-8B multimodal backbones, question-relevant historical frames can be retrieved from the keys of only the first few language-Transformer layers (layer ~3-4 of 28-32), so continuous frames need not be prefilled through full model depth: indexing shallow and re-prefilling only the retrieved units at full depth cuts per-frame prefill up to 52x and 20-second end-to-end latency up to 15x while scoring about 1.7 points above the best same-backbone baseline on OVO-Bench and StreamingBench.Reasoning about time in video · reviewed by AI
- In training-free video temporal grounding that localizes segments by cosine similarity between a text query embedding and video features, clip-level features alone localize worse than frame-level features alone, yet projecting clip similarities onto the frame timeline and mixing the two streams beats either alone, because frames supply boundary resolution and clips supply temporal context.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: An ablation on the same benchmarks where appending the retrieved evidence directly matches or beats the evolved latent memory tokens, or where the backward-tracing gain does not exceed the real-time perception gain. Proposed technique, not catalogued: latent working memory internalization for streaming video.