vlm-agents-must-verify-claims-against-interactive-visualizations-whereWhen VLM agents must verify claims against interactive visualizations where no claim is answerable from the initial viewport, giving them an interaction budget of ten actions does not reliably beat answering immediately from the first screenshot — some models score slightly lower with interaction — because partial, unplanned exploration leaves the model with enough evidence to abandon its prior but not enough to replace it.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Checking claims against evidence
Observed on
Screenshot-only agents (no DOM), multi-view Vega-Lite notebooks, three-way True/False/NEI labels, 500-claim stratified subset; ablation run at budgets of 1, 10 and 25 actions for o.
Sources
- Measured on 500 claims across fourteen model configurations. Gemini 3.5 Flash: 44.2% at one action, 41.6% at ten, 50.0% at twenty-five; Qwen 3.5 27B 46.2% vs 45.0%; Gemma 4 31B 45.4% vs 47.2%. Best overall was GPT-5.5 at 57.2% against a 33.3% chance baseline. The budget ablation covers only three models, and gaps are within a few points, so the dip is suggestive rather than established.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- Used zero-shot as step-level action critics in long-horizon tool-calling tasks, frontier models are over-pessimistic — flagging a large share of correct actions and pushing the actor into revision loops that lower success below no critic at all — whereas small models fine-tuned on action-level verification rationales flag far fewer valid actions and raise repeated-run reliability.Using the tools it is given · unreviewed
- In multi-step visual QA, a critic step run by the same model improves final accuracy only when it is shown the candidate evidence alone and denied the generator's textual rationale; conditioned on that rationale, the critic ratifies the generator's initial hypothesis and the loop performs worse than no critic at all.Fixing its own mistakes · unreviewed
- In partially observable text environments (ALFWorld, ScienceWorld), letting an LLM agent query an externally maintained state estimate that keeps an explicit distribution over unobserved object locations raises task success more than querying a deterministic memory of what has already been observed, and the gain shrinks to near zero on a frontier model that already nearly saturates the benchmark.Tracking state through a long task · unreviewed
- Tool-augmented multimodal models trained only on how to call vision tools accept tool outputs uncritically; adding supervised trajectories where tool evidence is checked against the image and re-queried, plus a reward for correction that reaches a correct answer, raises accuracy on perception-heavy benchmarks (counting, depth, spatial relations) in 3B and 7B Qwen2.5-VL, while slightly lowering it on knowledge-heavy ones.Fixing its own mistakes · unreviewed
- When a calculator-using model is trained with reinforcement learning on verifiable final-answer rewards for an arithmetic search task (Countdown), the gain lands almost entirely at low k — pass@1 rises sharply while pass@16 moves little or falls — because the update can only reinforce correct trajectories the starting policy already samples, and prompt groups with no correct sample supply no gradient.Using the tools it is given · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A comparable evaluation where the same models gain consistently in accuracy as soon as any interaction is allowed, with no non-monotonic dip at intermediate budgets. Proposed technique, not catalogued: Interaction Efficiency Score (accuracy weighted by fraction of actions that change the observed state).