arxiv-2608-25529 · paper

Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios

Created: 2026-08-26 · Ingested: 2026-09-09

Multimodal Large Language Models (MLLMs) have shown strong performance in video understanding. However, their ability to follow instructions in this domain remains under-explored. Real-world video understanding requires models not only to interpret video content correctly, but also to satisfy diverse user-specified constraints. Existing benchmarks focus primarily on task accuracy rather than instruction adherence, leaving this capability insufficiently evaluated. To address this gap, we introduce Video-IFBench, a comprehensive benchmark for evaluating instruction following in video understandi

In brief

Video multimodal models fail at instruction following in a way that scales with constraint count and conditional nesting, not with task count. Handling several tasks in one request is close to free; satisfying many constraints or picking the right video-grounded branch is not.

Video-IFBench pairs 1.5K samples over more than 700 videos (10 seconds to 10 minutes, approximately 49 hours total) with 4 instruction templates (Single, Multi, Selection, Nested), 32 task types and 39 constraint types (22 semantic, 17 format). Scoring is checklist-based, mixing an LLM judge with programmatic verifiers, gated on whether the response addresses the active task.

Across more than 20 models, the best is Gemini-3-Pro at 76.5 TCSR and 54.5 TISR; the best open-source model, Qwen3.5-397B-A17B-Think, reaches 69.6 and 46.1. Nested is worst: Gemini-3-Pro gets 53.7 TCSR. Task pass rate is flat across 1-4 to 9+ tasks (Gemini-3-Pro 95.6 to 96.3), while TISR falls with constraints (62.7 to 46.9 for Gemini-3-Pro, 40.1 to 3.5 for Qwen3-Omni-30B-A3B-Instruct).

Controlled ablations isolate branch count and branch position on a 3-condition-plus-else subset: Qwen3.5-397B-A17B-Instruct drops from 70.6% to 29.4% when the correct branch moves from position 1 to 3. A second judge model gave consistent trends. Frame sampling differs by model (1 FPS versus 50 frames for GPT-5.4), and vision-only models get transcribed subtitles, so cross-model comparison is not input-matched.

If you evaluate video models on answer correctness only, this says nothing about whether they will obey your formatting and conditional requirements.

Written from the abstract by claude-opus-5 on 2026-09-09, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsWhen a video-understanding instruction contains several conditional branches and the model must pick the branch matching what the video shows, both proprietary and open multimodal models pick correctly far more often when the correct branch is listed first, and accuracy falls as the correct branch moves later in the list.