arxiv-2608-20814 SRC-0104 · paper

Enhancing Localized Reasoning for Long Video Understanding via Efficient Segment-to-Video Supervision

Created: 2026-08-21 · Ingested: 2026-09-15

https://arxiv.org/abs/2608.20814

Though Multimodal Large Language Models (MLLMs) have shown impressive potential in video understanding, long video understanding (LVU) remains challenging since distracting noise in complex and lengthy contexts can obscure localized details, misleading MLLMs to produce incorrect answers. Recent works mitigate these issues by incentivizing deep reasoning to include relevant evidence. However, these methods have two main problems: First, the reinforcement fine-tuning framework (RFT) they leveraged incurs substantial training overheads, including high annotation costs and complicated reward desig

In brief

Generating video QA from short segments and then training on the full video, with segment timestamps in the prompt as training-only privileged information, improves long-video accuracy more than whole-video-annotated QA at the same data scale.

S2V builds a 10K-sample dataset from YT-Temporal-180M via scene detection, VQA generation with Qwen3-VL-235B, cross-checking with GLM-4.5V, and difficulty filtering with Qwen3-VL-8B. Training is GRPO with a single answer-accuracy reward on Qwen3-VL-4B, MiMo-VL-7B, and Qwen3-VL-8B, evaluated on LongVideoBench, Video-MME, and MLVU.

S2V-4B reaches 62.3 on LongVideoBench and 71.9 on MLVU against 56.2 and 66.1 for its base model, and beats reasoning-based methods trained on far more data (Video-R1: 425K samples, 58.5 LVB) while cutting GPU hours from roughly 440 to 160. Gains concentrate on the medium and long subsets and on MLVU's needle and counting splits.

The design isolates the cause reasonably well: an ablation separates whole-video VQA (V2V, 57.7 LVB) from segment VQA without timestamps (60.6) from full S2V (62.3), plus a frame-count sweep and a difficulty-ratio sweep. A held-out S2V-Test of 800 samples shows the segment-vs-whole-video accuracy gap shrinking after training. Data quality is self-reported: 86.7% model recheck pass rate, 93.0% on 500 manually inspected samples.

Only 4B-8B open models were tested; grounding accuracy itself barely improves, and 4B stops benefiting as data scales while 8B keeps improving.

If you are building long-video training data, annotating segments rather than whole videos may matter more than volume.

Written from the abstract by claude-opus-5 on 2026-09-15, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsFor 4-8B multimodal models on long-video QA, RL training on questions authored from short scene segments but answered over the whole video improves benchmark accuracy more than RL on questions authored from the whole video, because segment-authored questions target details a whole-video view overlooks; including the segment timestamp in the training instruction adds a further gain by removing referential ambiguity, and is not needed at inference.