video-understanding-instruction-contains-several-conditional-branches-model-must-pick CLM-0189observationsingle paperReviewed by AI
When a video-understanding instruction contains several conditional branches and the model must pick the branch matching what the video shows, both proprietary and open multimodal models pick correctly far more often when the correct branch is listed first, and accuracy falls as the correct branch moves later in the list.
Capability: Following an unfamiliar procedure
Observed on
Multimodal LLMs given video plus a selection-style instruction with three conditions plus an else branch; measured as task pass rate with the branch set held fixed and only the cor.
Sources
- supportsVideo-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding ScenariosControlled ablation on one subset of a new 1.5K-sample benchmark: same conditions, correct branch moved to position 1, 2 or 3, seven representative models. Drops are large for some (one model from 70.6 to 29.4) and one model does not follow the pattern. Single benchmark, single video-instruction format; no test of whether the bias is representational or decoding-level.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When a document image states multi-step selection constraints in narrative text and the supporting numbers sit in several charts, multimodal models degrade steadily as either the number of logical hops (2 to 5) or the number of charts (2 to 6) grows, with the best model reaching 62.83% overall against over 90% for human annotators.Carrying a chain of inference through to its conclusion · reviewed by AI
- When vision-language models are given an explicit abstention option plus a stated payoff for correct, wrong and declined answers, quadrupling the penalty for a wrong answer barely changes how often they answer, so their abstention rate is largely insensitive to the stated cost of error across text, synthetic video and natural video recall tasks.Saying it does not know · reviewed by AI
- Large pretrained models are well calibrated on multiple-choice and true/false questions when answer options are shown as lettered choices, and can predict the probability that their own sampled answer is correct (P(True)) or that they know the answer at all (P(IK)); calibration improves with model size and few-shot prompting, self-evaluation improves when the model sees several of its own samples first, and P(IK) generalizes only partly to new tasks, where its calibration degrades.Saying it does not know · reviewed by AI
- For 4-8B multimodal models on long-video QA, RL training on questions authored from short scene segments but answered over the whole video improves benchmark accuracy more than RL on questions authored from the whole video, because segment-authored questions target details a whole-video view overlooks; including the segment timestamp in the training instruction adds a further gain by removing referential ambiguity, and is not needed at inference.Reasoning about time in video · reviewed by AI
- When Video-LLMs are asked to verify sentence by sentence whether a dense video caption is supported by the video, they detect hallucinated sentences less often in captions they generated themselves than in captions from other models, and detection of temporal errors (action, order, causality, frequency, state change) improves much less with model scale than detection of static-frame errors.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Running the same controlled position swap on a broader set of models and finding pass rate flat across branch positions, or higher for later positions.