Reasoning about time in video

Claims

• observationsingle paper↑1↓0
  For 2B-8B video multimodal models, adding a small fraction of time-reversed video rollouts into each GRPO group (e.g. 1 reversed out of 7) raises accuracy on action and temporal-reasoning video QA over standard GRPO, while raising the reversed share further (2 or 3 of 7) gives back the gain.
• observationreplicated↑1↓0
  For 2B-8B video multimodal models fine-tuned for temporal reasoning, placing a small minority of reversed-video rollouts in each GRPO group (1 reversed out of 7) beats both all-forward GRPO and larger reversed shares (2 or 3 of 7), because more reversed samples dilute the main forward-video answering objective.
• mechanismsingle paper↑1↓0
  For 4-8B multimodal models on long-video QA, RL training on questions authored from short scene segments but answered over the whole video improves benchmark accuracy more than RL on questions authored from the whole video, because segment-authored questions target details a whole-video view overlooks; including the segment timestamp in the training instruction adds a further gain by removing referential ambiguity, and is not needed at inference.
• mechanismsingle paper↑1↓0
  For 4B-8B Qwen-VL video models post-trained by on-policy self-distillation, conditioning the frozen self-teacher only on the annotated evidence frames (while the student sees the full video) yields a more accurate teacher and better student scores than giving the teacher the same frames plus non-evidence context, and teacher accuracy falls monotonically as temporally irrelevant frames are added.
• observationsingle paper↑1↓0
  For 7-8B multimodal models answering multiple-choice long-video questions under a fixed 16-frame budget, selecting keyframes by embedding similarity to the query concatenated with each candidate answer beats query-only scoring, query-agnostic diversity selection, and uniform sampling across Qwen2-VL, LLaVA-Video and InternVL3.5, with the largest margins on the longest video splits — but per-category results still invert, including uniform sampling winning some categories.
• observationsingle paper↑1↓0
  For 7B multimodal models on long-video QA, a multi-turn agentic exploration harness (adaptive viewing speed, cross-turn textual notes, rewatch) lowers accuracy below plain uniform-frame single-pass answering unless the model is trained to use it, because the untrained model commits to an answer before gathering evidence; with cold-start SFT on verified exploration turns plus trajectory-level RL, the same harness beats the single-pass baseline.
• observationsingle paper↑1↓0
  On hour-scale (mean 88.8 min) open-ended streaming video QA with causal access, every streaming memory design degrades as the evidence-to-query gap grows, and they fail in different ways: recent-window systems cannot recover distant events (AURA 25.4% within 5 minutes falling to 10.5% beyond 30 minutes), text-summary systems keep temporal coverage but discard visual evidence, and model-internal compression loses fine-grained detail (StreamForest 14.4%, ThinkStream 7.5% on retrospection), while human accuracy without rewatching is 63.4%.
• observationsingle paper↑1↓0
  On long-video (10-150 minute) audio-visual reasoning questions, multiple-choice accuracy overstates grounding: a large open model can answer about 55% of items correctly while scoring 8.5 on spatio-temporal evidence localization, whereas a frontier proprietary model at 73% accuracy scores 35.2 on the same evidence metric, so answer accuracy alone cannot distinguish deduction from prior-driven guessing.
• mechanismsingle paper↑1↓0
  For long-video QA with a fixed frame budget fed to a frozen multimodal model, selecting frames by marginal contribution to the already-selected subset (query relevance plus coverage gain minus redundancy) beats ranking frames independently by query-frame similarity, and the margin over uniform sampling grows with video duration.
• observationsingle paper↑1↓0
  For long-video QA under a fixed 32-frame budget with 7B-8B multimodal backbones, selecting frames from the structural shape of the frame-query similarity curve (peak, rising, falling, transition regions) beats ranking frames by similarity magnitude, and the margin grows with video length — a few points on short clips and roughly 8 points on hour-long videos — while on short/medium-clip benchmarks it is near-indistinguishable from other selectors.
• observationsingle paper↑1↓0
  For long-video QA under a fixed frame budget with a frozen 7B multimodal model, organizing frames into a question-agnostic hierarchical temporal tree and searching it coarse-to-fine (answer, then issue a search statement, then expand the relevant branch) raises accuracy over uniform sampling and over agent methods that score frames independently, with the gain largest on the longest-video benchmarks.
• observationsingle paper↑1↓0
  On month-level egocentric recordings (20-120 days per participant) with human-written multiple-choice questions, current multimodal models degrade as the question moves from inferring recurring habits, to retrieving one specific episode, to composing evidence across days — the best model reached 71.8% macro-average against a 94.2% human baseline, with several models near or below the 25% chance level on route reasoning, cross-view spatial reasoning and direction judgement.
• observationsingle paper↑1↓0
  On multi-hop video questions written so no single frame or timestamp lookup suffices, single-turn MLLMs answer poorly (0.07-0.51 accuracy across 12 proprietary and open models), and wrapping the same models in an agentic loop helps only when the loop lets the agent choose what, where and at what granularity to look: frameworks giving adaptive fps/resolution/interval control raised accuracy and trajectory scores for nearly every model, while a fixed-schedule framework with weak perception tools (STAR) fell below the single-turn baseline for all of them.
• observationsingle paper↑1↓0
  For multi-task procedural video understanding with a small (1B) video-LLM decoder, replacing each decoder feed-forward block with task-specific expert copies and routing each video-prompt sample by its known task identity gives higher average top-1 accuracy across the five COIN tasks than the same expert topology run densely or with learned token-level or sample-level routing, with the largest gains on the procedure-forecasting tasks.
• observationsingle paper↑1↓0
  For streaming video question answering with a frozen video-LLM, appending retrieved historical frames as extra visual context helps less than optimizing a small set of latent memory tokens at test time and then dropping the retrieved tokens before decoding; the gain is largest on questions requiring backward tracing over past history.
• mechanismsingle paper↑1↓0
  For streaming video understanding with 7-8B multimodal backbones, question-relevant historical frames can be retrieved from the keys of only the first few language-Transformer layers (layer ~3-4 of 28-32), so continuous frames need not be prefilled through full model depth: indexing shallow and re-prefilling only the retrieved units at full depth cuts per-frame prefill up to 52x and 20-second end-to-end latency up to 15x while scoring about 1.7 points above the best same-backbone baseline on OVO-Bench and StreamingBench.
• mechanismsingle paper↑1↓0
  In training-free video temporal grounding with frozen vision-language embedding backbones, matching a query only against frames sampled at a fixed rate leaves temporal context unused: adding clip-level (sliding-window) similarities projected onto the frame timeline raises localization accuracy even though clip-only similarity is worse than frame-only on its own.
• mechanismsingle paper↑1↓0
  In training-free video temporal grounding that localizes segments by cosine similarity between a text query embedding and video features, clip-level features alone localize worse than frame-level features alone, yet projecting clip similarities onto the frame timeline and mixing the two streams beats either alone, because frames supply boundary resolution and clips supply temporal context.
• observationsingle paper↑1↓0
  When Video-LLMs are asked to verify sentence by sentence whether a dense video caption is supported by the video, they detect hallucinated sentences less often in captions they generated themselves than in captions from other models, and detection of temporal errors (action, order, causality, frequency, state change) improves much less with model scale than detection of static-frame errors.

Techniques

• Condition reasoning credit on spatio-temporal evidenceprocess
  Require the model to output timestamped audio and visual evidence intervals and multiply its answer correctness by a soft temporal-plus-semantic evidence score, so ungrounded correct answers earn no evidence credit.
• Condition the self-teacher only on evidence framestraining
  In on-policy self-distillation for video LLMs, the frozen self-teacher replays the student's trajectory while seeing only frames from the annotated evidence interval plus the gold answer, and token-level distillation is weighted by each tok
• Fuse frame- and clip-level similarity, then adapt the query decoding
  Compute query-video cosine similarities at both frame and sliding-window clip granularity, project clip scores onto the frame timeline with a triangular kernel and mix them with a fixed weight, and separately optimize a single additive offs
• Index the stream with shallow-layer KVs, answer deep only onarchitecture
  Streaming frames pass through only the shallow language layers whose KVs form a full-history index; at query time a text-only logit gate decides whether history is needed, cross-layer token voting plus max-min diversity picks evidence units
• Mix reversed-video rollouts into GRPO groupstraining
  During RL post-training of a video model, each group of sampled responses contains mostly forward-video rollouts plus one time-reversed rollout of the same clip, so correctness reward forces the policy to use temporal direction.
• ReAct loop over controllable video inspection toolstooling
  A single agent iteratively calls video observation tools where it sets the time interval, frame rate and resolution itself, judging evidence sufficiency each turn before answering.
• Route decoder FFN experts by task identity, not tokensarchitecture
  Convert each LLM decoder feed-forward network into one always-active shared expert plus one expert per task, and select a single task expert for the whole video-prompt sample using the task label or nearest-neighbour matching of the prompt
• Search a hierarchical temporal tree of video segments coarseretrieval
  Build a question-agnostic tree of contiguous video segments by recursive temporally constrained clustering with one key frame per node, then run an answer-retrieve-explore loop that expands only branches the model says it needs.
• Select frames by the shape of the relevance curve, not its hretrieval
  Smooths the frame-query similarity sequence, segments it by slope and curvature into peak, rising, falling, boundary and background regions, allocates the frame budget across regions using a question-derived preference vector, and picks fra
• Split streaming video into a latency frontend and an asynchrarchitecture
  Frontend workers handle dispatch and proactive monitoring while backend workers asynchronously build a hierarchical event and entity memory with retained key frames, and route recall and external search on demand.

Sources

Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.

• AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents2026-09-18
  ↑ supports a claim here
• VidOmni-Bench: A Benchmark for Fine-Grained Video Understanding via Spatio-Temporal Event Verification across Complexity and Duration2026-09-18
  ↑ supports a claim here
• Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?2026-09-15
  ↑ supports a claim here
• VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs2026-09-15
  about this capability · no claim drafted yet
• VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understanding2026-09-14
  ↑ supports a claim here
• MarKey: Marginal Utility Guided Greedy Keyframe Selection for Long Video Understanding2026-09-14
  ↑ supports a claim here
• DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding2026-09-08
  ↑ supports a claim here · ↑ supports a claim here
• Evaluation of MLLM-Agnostic Plug-and-Play Keyframe Selection Methods for Long Video Understanding2026-09-04
  ↑ supports a claim here
• Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding2026-09-03
  ↑ supports a claim here
• ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding2026-09-02
  ↑ supports a claim here
• ViTAL-X: Video-Text Alignment with Cross-Modal Temporal Edits2026-09-01
  about this capability · no claim drafted yet
• RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding2026-08-30
  ↑ supports a claim here
• Temporal Tree of Thought: Reasoning-Guided Visual Cue Search for Long-Video Understanding2026-08-28
  ↑ supports a claim here
• Video-OPSD: Exploiting Privileged Visual Evidence for On-Policy Self-Distillation in Video Large Language Models2026-08-27
  ↑ supports a claim here
• MoTE: Mixture of Task Experts for Multi-Task Video Understanding2026-08-25
  ↑ supports a claim here
• COMET: Contrastive Motion-Enhanced Temporal Reasoning for Video Multimodal Large Language Models2026-08-21
  ↑ supports a claim here · ↑ supports a claim here
• Enhancing Localized Reasoning for Long Video Understanding via Efficient Segment-to-Video Supervision2026-08-21
  ↑ supports a claim here
• EgoMonth: A Month-Level Egocentric Video Benchmark for Long-Term Spatiotemporal Memory2026-08-13
  ↑ supports a claim here
• StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understanding2026-08-06
  ↑ supports a claim here
• ChronoVision: Temporal Reasoning via Latent State Reconstruction2026-08-06
  about this capability · no claim drafted yet