tool-augmented-multimodal-models-trained-only-how-call-vision-toolsTool-augmented multimodal models trained only on how to call vision tools accept tool outputs uncritically; adding supervised trajectories where tool evidence is checked against the image and re-queried, plus a reward for correction that reaches a correct answer, raises accuracy on perception-heavy benchmarks (counting, depth, spatial relations) in 3B and 7B Qwen2.5-VL, while slightly lowering it on knowledge-heavy ones.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Fixing its own mistakes
Observed on
Qwen2.5-VL 3B and 7B backbones with a fixed suite of vision tools (detection, zoom, edge, depth); training trajectories synthesized by Gemini-2.5-Pro; gains do not transfer to MMMU.
Sources
- Measured on CountBench, CVBench, BLINK, BLINK-HARD, MMVP, MMSTAR and others against Qwen2.5-VL-Instruct, SFT, SFT-GRPO, and tool-augmented baselines (ReVPT, Thyme, CodeDance), with an ablation separating cold-start data, correction reward and grounding reward. Single group, margins over prior SoTA are small and mixed per benchmark; the trajectory data comes from a stronger proprietary model, so distillation is not isolated from the verification mechanism.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- When a multimodal model is given external text that contradicts the image, making it commit to a description of the image before the text is admitted, then reconciling the two in a second call, recovers accuracy the single-prompt setup loses — but the benefit of withholding the text from the first call is model- and text-source dependent, and on some models context-preserving prompting is still better.Telling the user what they want to hear · unreviewed
- On multi-turn tool-calling benchmarks that score whether the final environment state matches a gold trajectory, aggregate accuracy hides wrong action-class choices: a model can call a withheld or under-specified tool with fabricated arguments, be handed the missing piece on a later turn, and still be graded PASS — so heavily tool-trained model families rank high while almost never asking for a missing parameter or refusing an unavailable function.Whether the measurement made the finding · unreviewed
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · unreviewed
- In UI-to-code generation, where a rendered screenshot gives genuine external feedback, iterative self-critique still drifts below the initial draft over ten rounds, because a code edit changes the rendering non-locally and a fix for one region breaks regions that were already faithful; restricting each round to one typed, scoped repair target and carrying past targets forward as an avoid-list reduces that drift.Generating and editing working code · unreviewed
- In agent pipelines that generate whole domain-specific ML codebases (medical imaging) from scratch, execution/assembly validation and domain knowledge bases fix different failures: removing runtime and assembly testing mainly costs autonomy (more human interventions to debug), while removing the curated domain knowledge base mainly costs output quality (syntactically fine but domain-inappropriate pipelines) with little change in intervention count.Generating and editing working code · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An ablation showing the curated verification trajectories and correction reward give no gain over standard tool-use SFT plus accuracy-only GRPO on the same perception benchmarks and backbone. Drafted stance toward external-feedback-repair-works-only-with-real-grounding: supports -- The correction loop works because the model checks tool output against the image itself, an external perceptual signal, and the paper notes recovery fails when that signal is ambiguous or the tool fails outright. Proposed technique, not catalogued: verification-supervised tool self-correction training.