interactive-video-world-models-world-state-communicated-only-through CLM-0407In interactive video world models, world state communicated only through prompt switching is not reliably preserved across a rollout — characters that should stay alive vanish and killed ones keep moving — whereas keeping the canonical state in an external deterministic engine and compiling it into pixel-aligned spatial controls for the video model keeps the rendered frames consistent with that state.
Capability: Tracking state through a long task
Observed on
Game-style combat scenes rendered from a single reference frame; evaluated on 50 short clips with a VLM judge counting visibly alive characters and checking death events..
Sources
- supportsProgrammable World ModelValidated by: On CombatStateBench (50 clips), 94% Count Accuracy and 98% State Accuracy versus 40.75/8.00 for LingBot-World-V2 and 32.00/58.00 for YUME under prompt switching, plus better VBench consistency scores; judged by a VLM given only RGB frames.. Measured on one self-built 50-clip benchmark against two baselines that lack an instance-level state interface, so the comparison is partly an interface mismatch rather than an isolated test of state externalization; metrics come from a VLM judge and are coarse (counts, any-frame death depiction).
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In training-free video temporal grounding with frozen vision-language embedding backbones, matching a query only against frames sampled at a fixed rate leaves temporal context unused: adding clip-level (sliding-window) similarities projected onto the frame timeline raises localization accuracy even though clip-only similarity is worse than frame-only on its own.Reasoning about time in video · reviewed by AI
- When a model must anticipate, without execution feedback, what a change to its own runtime harness components will do, naming which components a lifecycle change can affect stays near ceiling as the number of interacting effects grows from 2 to 32, while predicting the resulting state, reasoning across teardown orders, and choosing minimal reconfigurations degrade sharply (e.g. GPT-5.6 Luna's reachable-condition Jaccard 91.7% to 14.1%, executed reconfiguration success 62.5% to 25.0%).Improving its own scaffolding from failures · reviewed by AI
- When Video-LLMs are asked to verify sentence by sentence whether a dense video caption is supported by the video, they detect hallucinated sentences less often in captions they generated themselves than in captions from other models, and detection of temporal errors (action, order, causality, frequency, state change) improves much less with model scale than detection of static-frame errors.Reasoning about time in video · reviewed by AI
- When a video-understanding instruction contains several conditional branches and the model must pick the branch matching what the video shows, both proprietary and open multimodal models pick correctly far more often when the correct branch is listed first, and accuracy falls as the correct branch moves later in the list.Following an unfamiliar procedure · reviewed by AI
- On multi-hop video questions written so no single frame or timestamp lookup suffices, single-turn MLLMs answer poorly (0.07-0.51 accuracy across 12 proprietary and open models), and wrapping the same models in an agentic loop helps only when the loop lets the agent choose what, where and at what granularity to look: frameworks giving adaptive fps/resolution/interval control raised accuracy and trajectory scores for nearly every model, while a fixed-schedule framework with weak perception tools (STAR) fell below the single-turn baseline for all of them.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An interactive video world model with no external state engine matching ~94% count and ~98% state accuracy on the same benchmark, or the engine-conditioned system losing state consistency on longer rollouts. Drafted stance toward models-predict-next-steps-well-can-still-hold: supports -- Baselines generate visually plausible next frames while holding an incoherent implicit world state that breaks on entity counts and persistent death states.