vlm-agents-must-verify-claims-against-interactive-visualizations-where CLM-0190When VLM agents must verify claims against interactive visualizations where no claim is answerable from the initial viewport, giving them an interaction budget of ten actions does not reliably beat answering immediately from the first screenshot — some models score slightly lower with interaction — because partial, unplanned exploration leaves the model with enough evidence to abandon its prior but not enough to replace it.
Capability: Checking claims against evidence
Observed on
Screenshot-only agents (no DOM), multi-view Vega-Lite notebooks, three-way True/False/NEI labels, 500-claim stratified subset; ablation run at budgets of 1, 10 and 25 actions for o.
Sources
- Measured on 500 claims across fourteen model configurations. Gemini 3.5 Flash: 44.2% at one action, 41.6% at ten, 50.0% at twenty-five; Qwen 3.5 27B 46.2% vs 45.0%; Gemma 4 31B 45.4% vs 47.2%. Best overall was GPT-5.5 at 57.2% against a 33.3% chance baseline. The budget ablation covers only three models, and gaps are within a few points, so the dip is suggestive rather than established.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When VLMs must operate unfamiliar household devices with no manual, through multi-turn interaction over rendered multi-view images with click/rotate actions at pixel coordinates, the limiting failure is fine-grained localization of the interactive component within the correct view rather than navigating to that view, and almost no task is completed in the optimal number of steps.Turning plans into physical action · reviewed by AI
- For post-training a pretrained vision-language-action manipulation policy under a fixed per-round frame budget, collecting handheld demonstrations only at states where the human action chunk diverges from the policy's sampled action distribution, and weighting them by a learned task-progress advantage, raises task progress more than collecting the same number of frames of ordinary demonstrations and fine-tuning on them.Turning plans into physical action · reviewed by AI
- When open-source vision-language models are post-trained with RL or critique tuning to act as judges of chart-to-code outputs, they become more likely to output Accept rather than better at verification: pairwise-comparison accuracy barely improves under an order-swapped protocol, while acceptance rates rise and specificity falls, so they ratify flawed but plausible-looking charts.Judging another's work as well as a careful reviewer would · reviewed by AI
- For deciding when a robot skill has finished and the next subtask should begin, general-purpose VLMs prompted zero-shot are near useless on long-horizon mobile manipulation (under 4% of transitions detected on BEHAVIOR-1K, including a frontier proprietary model), while a sub-1B student fine-tuned on compact self-explanations condensed from teacher reasoning over annotated demonstrations detects about 90% and lifts closed-loop task success from 0.30 to 0.53 against a 0.63 oracle.Turning plans into physical action · reviewed by AI
- In VLM-driven human-aware robot task planning over scene graphs, the bottleneck is inferring and grounding the symbolic goal, not the planning: once both human and robot goals correctly cover the ground truth, a formal multi-agent planner produces a valid, conflict-free joint plan in 99.6% of trials, while end-to-end success with VLM-inferred human goals is 69.7%.Planning a sequence of actions before acting · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A comparable evaluation where the same models gain consistently in accuracy as soon as any interaction is allowed, with no non-monotonic dip at intermediate budgets. Proposed technique, not catalogued: Interaction Efficiency Score (accuracy weighted by fraction of actions that change the observed state).