vlm-based-judging-camera-conditioned-world-model-videos-splitting-long-video-pair CLM-0656For VLM-based judging of camera-conditioned world-model videos, splitting a long video pair into action-aligned chunks with structured visual evidence (source image, frame-grid overview, per-action first/last-frame panels) and voting chunk verdicts into video-level preferences agrees with human preferences better than judging the whole video pair at once, because short-lived action evidence is otherwise diluted by the full context.
Capability: Judging another's work as well as a careful reviewer would
Observed on
Pairwise preference judging of camera-trajectory-conditioned generated videos sharing source image, caption and trajectory; 9B VLM fine-tuned on distilled and audited chunk-level r.
Sources
- Validated by: Three-way agreement (incl. Tie) with majority human labels on 760 pairs: 77.63/81.32/73.03 on action/appearance/motion vs GPT-5.5 at 74.21/79.87/69.47 and Gemini 3.1 Pro at 65.79/80.13/60.79; ablations removing source image, frame grid or action panels each lower average agreement by about 1.3-2.5 points; used as RL reward it beat WorldCompass's split geometry+image rewards on action adherence and HPSv3 in all six horizon settings.. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On adversarial video hallucination pairs scored only when a method both accepts the factual statement and rejects the counterfactual one, purpose-built video hallucination detectors (embedding-similarity, learned-entailment, atomic-fact and attention-based) top out around 34.63% overall accuracy, well below a general frontier video-language model prompted to judge the same items (about 83.63%).Stating false facts confidently · reviewed by AI
- For 4-8B multimodal models on long-video QA, RL training on questions authored from short scene segments but answered over the whole video improves benchmark accuracy more than RL on questions authored from the whole video, because segment-authored questions target details a whole-video view overlooks; including the segment timestamp in the training instruction adds a further gain by removing referential ambiguity, and is not needed at inference.Reasoning about time in video · reviewed by AI
- When open-source vision-language models are post-trained with RL or critique tuning to act as judges of chart-to-code outputs, they become more likely to output Accept rather than better at verification: pairwise-comparison accuracy barely improves under an order-swapped protocol, while acceptance rates rise and specificity falls, so they ratify flawed but plausible-looking charts.Judging another's work as well as a careful reviewer would · reviewed by AI
- On long-video (10-150 minute) audio-visual reasoning questions, multiple-choice accuracy overstates grounding: a large open model can answer about 55% of items correctly while scoring 8.5 on spatio-temporal evidence localization, whereas a frontier proprietary model at 73% accuracy scores 35.2 on the same evidence metric, so answer accuracy alone cannot distinguish deduction from prior-driven guessing.Reasoning about time in video · reviewed by AI
- For 4B-8B Qwen-VL video models post-trained by on-policy self-distillation, conditioning the frozen self-teacher only on the annotated evidence frames (while the student sees the full video) yields a more accurate teacher and better student scores than giving the teacher the same frames plus non-evidence context, and teacher accuracy falls monotonically as temporally irrelevant frames are added.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A whole-video VLM judge given the same frames and the same reasoning-augmented training data matches or beats the chunked-and-voted judge on human agreement, or removing the chunk decomposition while keeping the training data leaves agreement unchanged. Automatic check flagged: figures not in the source: 7.25.