multi-task-procedural-video-understanding-small-1b-video-llm-decoder-replacing CLM-0453For multi-task procedural video understanding with a small (1B) video-LLM decoder, replacing each decoder feed-forward block with task-specific expert copies and routing each video-prompt sample by its known task identity gives higher average top-1 accuracy across the five COIN tasks than the same expert topology run densely or with learned token-level or sample-level routing, with the largest gains on the procedure-forecasting tasks.
Capability: Reasoning about time in video
Observed on
Five COIN procedural video tasks (step recognition, next-step forecasting, task recognition, two procedure-forecasting variants) with a Llama-3.2-1B decoder and SigLIP2 encoder; ta.
Sources
- Validated by: COIN five-task average top-1 accuracy: 62.9% for task-index routing versus 61.5% dense all-expert activation, 61.4% learned token top-1, and 62.1% learned sample top-1, all under the same five-expert topology and backbone; also above matched 1B and 3B dense VideoLLM-online baselines (59.3, 59.9) and reported 8B baselines (61.8), while activating ~2B LLM parameters per sample.. Measured on one dataset (COIN) with a single 1B backbone and one training run per condition; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For a same-family pair of reasoning and instruction-tuned 4B models, learning one interpolation ratio per module (attention block, FFN block, embedding, final norm) with a reward that scores correctness and ranks correct responses by length cuts average generated tokens by about 27% relative to the reasoning model while holding average accuracy, and beats fixed global-ratio merging and CMA-ES search over the same ratios.Cutting to the chase · reviewed by AI
- For 4-8B multimodal models on long-video QA, RL training on questions authored from short scene segments but answered over the whole video improves benchmark accuracy more than RL on questions authored from the whole video, because segment-authored questions target details a whole-video view overlooks; including the segment timestamp in the training instruction adds a further gain by removing referential ambiguity, and is not needed at inference.Reasoning about time in video · reviewed by AI
- For VLM-based judging of camera-conditioned world-model videos, splitting a long video pair into action-aligned chunks with structured visual evidence (source image, frame-grid overview, per-action first/last-frame panels) and voting chunk verdicts into video-level preferences agrees with human preferences better than judging the whole video pair at once, because short-lived action evidence is otherwise diluted by the full context.Judging another's work as well as a careful reviewer would · reviewed by AI
- On adversarial video hallucination pairs scored only when a method both accepts the factual statement and rejects the counterfactual one, purpose-built video hallucination detectors (embedding-similarity, learned-entailment, atomic-fact and attention-based) top out around 34.63% overall accuracy, well below a general frontier video-language model prompted to judge the same items (about 83.63%).Stating false facts confidently · reviewed by AI
- On multi-hop video questions written so no single frame or timestamp lookup suffices, single-turn MLLMs answer poorly (0.07-0.51 accuracy across 12 proprietary and open models), and wrapping the same models in an agentic loop helps only when the loop lets the agent choose what, where and at what granularity to look: frameworks giving adaptive fps/resolution/interval control raised accuracy and trajectory scores for nearly every model, while a fixed-schedule framework with weak perception tools (STAR) fell below the single-turn baseline for all of them.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Under the same expert topology and backbone, learned sample-level or token-level routing matching or beating explicit task-index routing on the COIN five-task average, or the gap vanishing once the dense baseline is trained to convergence with matched compute. Automatic check flagged: figures not in the source: 1.9.