multi-step-visual-qa-critic-step-run-same-model-improvesIn multi-step visual QA, a critic step run by the same model improves final accuracy only when it is shown the candidate evidence alone and denied the generator's textual rationale; conditioned on that rationale, the critic ratifies the generator's initial hypothesis and the loop performs worse than no critic at all.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Fixing its own mistakes
Observed on
Frontier VLMs (Gemini 2.5 Pro, Qwen3-VL, GPT-5, Grok) on high-resolution visual search benchmarks where the bottleneck is locating a small detail, not reasoning; critic and generat.
Sources
- Ablation on V* with Gemini 2.5 Pro: direct 83.8, locator only 89.2, locator+critic seeing crops only 92.7, locator+critic also seeing locator text 87.2; plus a token-matched baseline comparison (Explicit Locate-Verify-Answer 85.9). Single benchmark for the ablation, one backbone; the headline gains replicate across four VLMs and three benchmarks but the blinding ablation itself does not.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- When training small instruction-tuned models with reinforcement learning plus factual supervision, routing each atomic fact's verification score only to the tokens that produced it, and down-weighting verifier judgements that do not change when their key evidence is removed, improves factuality benchmark scores over trajectory-level or reasoning- step-level factual rewards.Checking claims against evidence · unreviewed
- Used zero-shot as step-level action critics in long-horizon tool-calling tasks, frontier models are over-pessimistic — flagging a large share of correct actions and pushing the actor into revision loops that lower success below no critic at all — whereas small models fine-tuned on action-level verification rationales flag far fewer valid actions and raise repeated-run reliability.Using the tools it is given · unreviewed
- In open-weight instruction-tuned models, interventions that suppress caving to user pushback (DPO, SFT on chosen responses, or activation steering) also tend to reduce the model's rate of correcting a wrong answer when genuine supporting evidence is supplied, because the two answer-flip behaviors run on overlapping MLP neurons and attention heads with positively aligned steering directions.Telling the user what they want to hear · unreviewed
- When a calculator-using model is trained with reinforcement learning on verifiable final-answer rewards for an arithmetic search task (Countdown), the gain lands almost entirely at low k — pass@1 rises sharply while pass@16 moves little or falls — because the update can only reinforce correct trajectories the starting policy already samples, and prompt groups with no correct sample supply no gradient.Using the tools it is given · unreviewed
- For training-free inference-time hallucination mitigation in 7B vision- language models, reported gains on object-hallucination benchmarks are largely inseparable from reduced informativeness: hallucination rate and object recall/coverage move together, so a lower score can mean the model mentioned fewer visual entities rather than grounded better.Whether the measurement made the finding · unreviewed
Notes
Ingested unreviewed on 2026-09-07 and deliberately inert until a human endorses it: it does not move a technique standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Drafted confidence: medium. Falsifier as drafted: An ablation on comparable visual-search tasks where feeding the locator's textual reasoning to the critic matches or beats the crops-only critic, or where critic-in-the-loop gains persist regardless of what the critic sees. Proposed technique, not yet catalogued: Critic blinded to the generator's rationale.