training-free-video-temporal-grounding-localizes-segments-cosine-similarity-between CLM-0229In training-free video temporal grounding that localizes segments by cosine similarity between a text query embedding and video features, clip-level features alone localize worse than frame-level features alone, yet projecting clip similarities onto the frame timeline and mixing the two streams beats either alone, because frames supply boundary resolution and clips supply temporal context.
Capability: Reasoning about time in video
Observed on
Similarity-matching pipelines with a frozen video-capable VLM embedding backbone (Qwen3-VL-Embedding-8B, 3 fps frames, 4s/2s clip windows), evaluated on Charades-STA and QVHighligh.
Sources
- Single ablation table on Charades-STA: frame-only 47.06 mIoU, clip-only 42.82, fused 51.30, with R@0.7 rising from 27.69/15.40 to 32.23. Query-side test-time adaptation contributes a smaller separate gain (45.69 to 47.06). One backbone family, two benchmarks; no isolation of kernel bandwidth or mixing weight choice, and the fusion weight is a tuned hyperparameter.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In training-free video temporal grounding with frozen vision-language embedding backbones, matching a query only against frames sampled at a fixed rate leaves temporal context unused: adding clip-level (sliding-window) similarities projected onto the frame timeline raises localization accuracy even though clip-only similarity is worse than frame-only on its own.Reasoning about time in video · reviewed by AI
- For long-video QA under a fixed 32-frame budget with 7B-8B multimodal backbones, selecting frames from the structural shape of the frame-query similarity curve (peak, rising, falling, transition regions) beats ranking frames by similarity magnitude, and the margin grows with video length — a few points on short clips and roughly 8 points on hour-long videos — while on short/medium-clip benchmarks it is near-indistinguishable from other selectors.Reasoning about time in video · reviewed by AI
- For streaming video question answering with a frozen video-LLM, appending retrieved historical frames as extra visual context helps less than optimizing a small set of latent memory tokens at test time and then dropping the retrieved tokens before decoding; the gain is largest on questions requiring backward tracing over past history.Reasoning about time in video · reviewed by AI
- For 4-8B multimodal models on long-video QA, RL training on questions authored from short scene segments but answered over the whole video improves benchmark accuracy more than RL on questions authored from the whole video, because segment-authored questions target details a whole-video view overlooks; including the segment timestamp in the training instruction adds a further gain by removing referential ambiguity, and is not needed at inference.Reasoning about time in video · reviewed by AI
- For long-video QA under a fixed frame budget with a frozen 7B multimodal model, organizing frames into a question-agnostic hierarchical temporal tree and searching it coarse-to-fine (answer, then issue a search statement, then expand the relevant branch) raises accuracy over uniform sampling and over agent methods that score frames independently, with the gain largest on the longest-video benchmarks.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An ablation on the same backbone where fused frame+clip similarity scores match or fall below the better single-scale stream, or where clip-only matches frame-only, would show the complementarity claim is wrong. Proposed technique, not catalogued: Multi-scale frame-plus-clip similarity fusion for temporal grounding.