video-understanding-instruction-contains-several-conditional-branches-model-must-pickWhen a video-understanding instruction contains several conditional branches and the model must pick the branch matching what the video shows, both proprietary and open multimodal models pick correctly far more often when the correct branch is listed first, and accuracy falls as the correct branch moves later in the list.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Following an unfamiliar procedure
Observed on
Multimodal LLMs given video plus a selection-style instruction with three conditions plus an else branch; measured as task pass rate with the branch set held fixed and only the cor.
Sources
- supportsVideo-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding ScenariosControlled ablation on one subset of a new 1.5K-sample benchmark: same conditions, correct branch moved to position 1, 2 or 3, seven representative models. Drops are large for some (one model from 70.6 to 29.4) and one model does not follow the pattern. Single benchmark, single video-instruction format; no test of whether the bias is representational or decoding-level.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- When a calculator-using model is trained with reinforcement learning on verifiable final-answer rewards for an arithmetic search task (Countdown), the gain lands almost entirely at low k — pass@1 rises sharply while pass@16 moves little or falls — because the update can only reinforce correct trajectories the starting policy already samples, and prompt groups with no correct sample supply no gradient.Using the tools it is given · unreviewed
- In open-ended editing tasks with no unique correct output (multi-slide design editing), a separate judge model that scores the structural diff between the original and edited artifact against the instruction, and returns its critique verbatim as the next-turn instruction, raises instruction-following over a single pass, but the gain is selective: most tasks pass on the first attempt, and a minority of the ones that enter the loop get worse unless an earlier iteration is restored when the judge's own score declines.Fixing its own mistakes · unreviewed
- Fine-tuning a small multimodal model on synthesized constraint-rich instructions raises adherence to output-level constraints (format, word count, keywords) while lowering accuracy on perception-grounded instructions, so the average can fall below the base model unless the synthesis loop tracks per-constraint failures and image compatibility.Following an unfamiliar procedure · unreviewed
- In multi-step visual QA, a critic step run by the same model improves final accuracy only when it is shown the candidate evidence alone and denied the generator's textual rationale; conditioned on that rationale, the critic ratifies the generator's initial hypothesis and the loop performs worse than no critic at all.Fixing its own mistakes · unreviewed
- When comparing a weight-shared recursive model against a depth-matched plain transformer on code generation, the ranking read from a single checkpoint can invert during training, because the larger-parameter arm passes its validation optimum and overfits while the parameter-lean recursive arm keeps improving — so a late-epoch "recursion fits better" reading is a degradation artifact, not a capability difference.Whether the measurement made the finding · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Running the same controlled position swap on a broader set of models and finding pass rate flat across branch positions, or higher for later positions.