multi-step-visual-qa-critic-step-run-same-model-improves CLM-0101In multi-step visual QA, a critic step run by the same model improves final accuracy only when it is shown the candidate evidence alone and denied the generator's textual rationale; conditioned on that rationale, the critic ratifies the generator's initial hypothesis and the loop performs worse than no critic at all.
Capability: Fixing its own mistakes
Observed on
Frontier VLMs (Gemini 2.5 Pro, Qwen3-VL, GPT-5, Grok) on high-resolution visual search benchmarks where the bottleneck is locating a small detail, not reasoning; critic and generat.
Sources
- Ablation on V* with Gemini 2.5 Pro: direct 83.8, locator only 89.2, locator+critic seeing crops only 92.7, locator+critic also seeing locator text 87.2; plus a token-matched baseline comparison (Explicit Locate-Verify-Answer 85.9). Single benchmark for the ablation, one backbone; the headline gains replicate across four VLMs and three benchmarks but the blinding ablation itself does not.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In closed-book multi-hop QA the accuracy gain from decomposed prompting shrinks to little in frontier models, but answers still change between task-equivalent prompt regimes (agreement only about 70%, and about 60% on MuSiQue), and because stable facts survive re-prompting while hallucinations do not, disagreement across regimes detects errors better than standard uncertainty baselines on F1 and AUROC without retrieval or training.Recalling facts it was trained on · reviewed by AI
- When training small instruction-tuned models with reinforcement learning plus factual supervision, routing each atomic fact's verification score only to the tokens that produced it, and down-weighting verifier judgements that do not change when their key evidence is removed, improves factuality benchmark scores over trajectory-level or reasoning- step-level factual rewards.Checking claims against evidence · reviewed by AI
- Used zero-shot as step-level action critics in long-horizon tool-calling tasks, frontier models are over-pessimistic — flagging a large share of correct actions and pushing the actor into revision loops that lower success below no critic at all — whereas small models fine-tuned on action-level verification rationales flag far fewer valid actions and raise repeated-run reliability.Using the tools it is given · reviewed by AI
- When automatically rewriting the prompts of a multi-agent LLM system from its failures, locating the responsible agent by injecting a hint into one agent at a time and using that agent's hint-corrected intermediate output as the supervision signal produces prompt updates that generalize better than critique derived only from the system-level output.Improving its own scaffolding from failures · reviewed by AI
- For a frozen LLM run in a generate-critique-revise loop with no external feedback, the accuracy gain is concentrated where single-shot reasoning is already unreliable: on clean multi-step reasoning (BBH, ~74% single-shot) depth makes no difference beyond the confidence interval, while on harder math (MATH, 26.2% single-shot) the same loop gains +14.2 points, with GSM8K in between (+4.2).Fixing its own mistakes · reviewed by AI
Notes
Ingested unreviewed on 2026-09-07 and deliberately inert until a human endorses it: it does not move a technique standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Drafted confidence: medium. Falsifier as drafted: An ablation on comparable visual-search tasks where feeding the locator's textual reasoning to the critic matches or beats the crops-only critic, or where critic-in-the-loop gains persist regardless of what the critic sees. Proposed technique, not yet catalogued: Critic blinded to the generator's rationale.