tool-augmented-multimodal-models-trained-only-how-call-vision-tools CLM-0168Tool-augmented multimodal models trained only on how to call vision tools accept tool outputs uncritically; adding supervised trajectories where tool evidence is checked against the image and re-queried, plus a reward for correction that reaches a correct answer, raises accuracy on perception-heavy benchmarks (counting, depth, spatial relations) in 3B and 7B Qwen2.5-VL, while slightly lowering it on knowledge-heavy ones.
Capability: Fixing its own mistakes
Observed on
Qwen2.5-VL 3B and 7B backbones with a fixed suite of vision tools (detection, zoom, edge, depth); training trajectories synthesized by Gemini-2.5-Pro; gains do not transfer to MMMU.
Applies when
- Effect on the capability:
- helps, narrowly
- Needs:
- access to train the weights
- Cost:
- high
- Fails when:
- The benchmark is knowledge-heavy rather than perception-heavy: accuracy slightly falls, and the gains do not transfer to MMMU.
Sources
- Measured on CountBench, CVBench, BLINK, BLINK-HARD, MMVP, MMSTAR and others against Qwen2.5-VL-Instruct, SFT, SFT-GRPO, and tool-augmented baselines (ReVPT, Thyme, CodeDance), with an ablation separating cold-start data, correction reward and grounding reward. Single group, margins over prior SoTA are small and mixed per benchmark; the trajectory data comes from a stronger proprietary model, so distillation is not isolated from the verification mechanism.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For 7-13B multimodal models, verifying each object claim with a frozen open-vocabulary detector lowers object hallucination only if evidence is pooled across semantically equivalent phrasings and checked for spatial agreement (the preferred boxes of different queries overlapping on one instance); a single-query maximum score is a weaker signal, and shuffling the spatial evidence across samples drives its discrimination to chance.Grounding answers in the image · reviewed by AI
- For 8B multimodal models, preference tuning and plain outcome-reward GRPO lower hallucination rates while making stated confidence worse calibrated than the untuned backbone, because rank- or correctness-based rewards never score the numerical accuracy of the model's own confidence; adding a Brier-plus-group-ECE calibration term to the GRPO reward cuts calibration error sharply and leaves accuracy metrics largely unchanged.Grounding answers in the image · reviewed by AI
- For single-step tool-calling agents, near-perfect clean accuracy does not predict robustness, and the dominant weak point is recovering the answer from perturbed tool returns rather than selecting the tool or binding arguments: across seven models, clean success averaged 0.979 while success under corrupted or missing returned evidence averaged 0.455, with no model above 0.60.Using the tools it is given · reviewed by AI
- For 4-8B multimodal models on long-video QA, RL training on questions authored from short scene segments but answered over the whole video improves benchmark accuracy more than RL on questions authored from the whole video, because segment-authored questions target details a whole-video view overlooks; including the segment timestamp in the training instruction adds a further gain by removing referential ambiguity, and is not needed at inference.Reasoning about time in video · reviewed by AI
- In RL-trained multimodal image-quality models, applying the reasoning reward only to reasoning tokens and the human-rating reward only to rating tokens keeps rating alignment at the level of rating-only training while turning reasoning-guided edit quality gain positive, and it prevents the collapse to a single universal edit that an unmasked shared reward produces despite that variant's larger raw gain.Saying what actually drove the answer · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An ablation showing the curated verification trajectories and correction reward give no gain over standard tool-use SFT plus accuracy-only GRPO on the same perception benchmarks and backbone. Drafted stance toward external-feedback-repair-works-only-with-real-grounding: supports -- The correction loop works because the model checks tool output against the image itself, an external perceptual signal, and the paper notes recovery fails when that signal is ambiguous or the tool fails outright. Proposed technique, not catalogued: verification-supervised tool self-correction training.