long-video-qa-frozen-vlm-fixed-answering-interface-outer-loop CLM-0604For long-video QA with a frozen VLM and fixed answering interface, an outer loop that proposes, executes and retains only development-accuracy-improving rewrites of the executable context-construction program can advance past a strong hand-crafted keyframe-selection harness, and the gain survives a matched cumulative visual-token control and direct reuse on other long-video benchmarks.
Capability: Improving its own scaffolding from failures
Observed on
One frozen 8B VLM (Qwen3-VL-8B-Instruct), per-call cap of 40 frames, ten search iterations, LVBench development split for promotion; one data seed, one proposer model, no search-va.
Sources
- Validated by: LVBench held-out accuracy (882 questions never used for promotion): AKS parent 46.6, searched CardinalityLedger 54.8, with paired bootstrap CI and exact McNemar; a capacity-matched AKS-90 control at similar cumulative visual tokens reached only 49.2; frozen transfer gave 67.5 vs 66.9 on Video-MME and 75.0 vs 72.0 on MLVU.. Measured on one benchmark for search plus two for transfer, with a cost-matched control and paired significance tests; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For long-video QA under a fixed frame budget with a frozen 7B multimodal model, organizing frames into a question-agnostic hierarchical temporal tree and searching it coarse-to-fine (answer, then issue a search statement, then expand the relevant branch) raises accuracy over uniform sampling and over agent methods that score frames independently, with the gain largest on the longest-video benchmarks.Reasoning about time in video · reviewed by AI
- For 4-8B multimodal models on long-video QA, RL training on questions authored from short scene segments but answered over the whole video improves benchmark accuracy more than RL on questions authored from the whole video, because segment-authored questions target details a whole-video view overlooks; including the segment timestamp in the training instruction adds a further gain by removing referential ambiguity, and is not needed at inference.Reasoning about time in video · reviewed by AI
- For 7-8B multimodal models answering multiple-choice long-video questions under a fixed 16-frame budget, selecting keyframes by embedding similarity to the query concatenated with each candidate answer beats query-only scoring, query-agnostic diversity selection, and uniform sampling across Qwen2-VL, LLaVA-Video and InternVL3.5, with the largest margins on the longest video splits — but per-category results still invert, including uniform sampling winning some categories.Reasoning about time in video · reviewed by AI
- For long-video QA under a fixed 32-frame budget with 7B-8B multimodal backbones, selecting frames from the structural shape of the frame-query similarity curve (peak, rising, falling, transition regions) beats ranking frames by similarity magnitude, and the margin grows with video length — a few points on short clips and roughly 8 points on hour-long videos — while on short/medium-clip benchmarks it is near-indistinguishable from other selectors.Reasoning about time in video · reviewed by AI
- For multi-task procedural video understanding with a small (1B) video-LLM decoder, replacing each decoder feed-forward block with task-specific expert copies and routing each video-prompt sample by its known task identity gives higher average top-1 accuracy across the five COIN tasks than the same expert topology run densely or with learned token-level or sample-level routing, with the largest gains on the procedure-forecasting tasks.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A replication with other seeds or VLMs where the searched harness fails to beat the AKS parent on held-out questions, or where the entire gain disappears once cumulative visual-token cost is matched. Drafted stance toward a-fixed-model-can-improve-its-own-harness-with-a-regression-gated-loop: supports -- Shows the same propose-evaluate-retain harness loop raising scores with weights frozen, here with a genuinely held-out split that was excluded from promotion.