used-zero-shot-step-level-action-critics-long-horizon-tool-calling-tasks-frontierUsed zero-shot as step-level action critics in long-horizon tool-calling tasks, frontier models are over-pessimistic — flagging a large share of correct actions and pushing the actor into revision loops that lower success below no critic at all — whereas small models fine-tuned on action-level verification rationales flag far fewer valid actions and raise repeated-run reliability.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Using the tools it is given
Observed on
tau-Bench/tau-Trait style multi-turn tool-calling domains (retail, airline, telecom, telehealth); actor fixed as Qwen3-32B; critics were GPT-4.1, Qwen3-235B, Qwen2.5-72B prompted, .
Sources
- Measured on four tool-calling domains with pass^1/pass^3/pass^4; reported GPT-4.1 critic flagging 46.8% of correct actions versus 13.6% and 11.4% for the trained 4B/8B critics, and average pass^4 falling from 11.5% (no critique) to 6.0% with GPT-4.1 critique while rising to 15.9%/19.5% with trained critics. Single paper, one model family for the trained critics, small out-of-domain task counts (18-20 tasks in telecom/telehealth), and prompting for the frontier critics was not tuned.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- General-purpose factuality verifiers (entailment-, consistency-, and RAG-trained) and zero-shot frontier LLM judges perform near chance at spotting unsupported claims in scientific peer reviews that must be checked against the full submitted paper, because they cannot separate paper-unsupported assertions from legitimate evaluative critique; small models fine-tuned on in-domain examples beat them by a wide margin.Stating false facts confidently · unreviewed
- When training small instruction-tuned models with reinforcement learning plus factual supervision, routing each atomic fact's verification score only to the tokens that produced it, and down-weighting verifier judgements that do not change when their key evidence is removed, improves factuality benchmark scores over trajectory-level or reasoning- step-level factual rewards.Checking claims against evidence · unreviewed
- In multi-step visual QA, a critic step run by the same model improves final accuracy only when it is shown the candidate evidence alone and denied the generator's textual rationale; conditioned on that rationale, the critic ratifies the generator's initial hypothesis and the loop performs worse than no critic at all.Fixing its own mistakes · unreviewed
- In open-ended editing tasks with no unique correct output (multi-slide design editing), a separate judge model that scores the structural diff between the original and edited artifact against the instruction, and returns its critique verbatim as the next-turn instruction, raises instruction-following over a single pass, but the gain is selective: most tasks pass on the first attempt, and a minority of the ones that enter the loop get worse unless an earlier iteration is restored when the judge's own score declines.Fixing its own mistakes · unreviewed
- In UI-to-code generation, where a rendered screenshot gives genuine external feedback, iterative self-critique still drifts below the initial draft over ten rounds, because a code edit changes the rendering non-locally and a fix for one region breaks regions that were already faithful; restricting each round to one typed, scoped repair target and carrying past targets forward as an avoid-list reduces that drift.Generating and editing working code · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A study where prompted frontier-model critics improve pass^k over no-critique baselines on the same benchmarks, or where their false-flag rate on valid actions matches that of small critics trained on verification rationales. Drafted stance toward self-critique-without-grounding-unreliable: supports -- Prompted frontier critics without training degraded actor performance relative to running with no critique agent, consistent with unaided critique being an unreliable improvement signal. Proposed technique, not catalogued: critique-aware supervision from action-level verification rationales.