arxiv-2608-30147 · paper

CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents

Created: 2026-08-31 · Ingested: 2026-09-09

Large language model (LLM) agents are increasingly deployed in long-horizon, interactive, and stateful environments. In these settings, a single wrong action, such as refunding the wrong purchase, can cause irreversible task failure and must be intercepted before execution. Such failures may not appear in every single run, but can emerge across repeated trials, making reliability across steps and trials critical. However, ensuring agentic reliability is challenging: even frontier LLMs struggle to explain why an action may be wrong, especially in long, intertwined trajectories governed by domai

In brief

Frontier models prompted as action-level critics for tool-calling agents are badly miscalibrated: GPT-4.1 flags 46.8% of correct actions, dragging the actor into revision loops and lowering success below no-critic baselines. Small critics fine-tuned on synthesized step-level rationales do better.

CAST samples 5 trajectories per task on the 500-task tau-Bench Retail split with Qwen3-32B as assistant, labels every action with a multi-agent verifier that sees privileged ground truth, trains Qwen3-4B/8B critics on the resulting 2,500 critique-enriched experiences, then fine-tunes a policy on successful critique-guided trajectories. Evaluation covers Retail (in-domain), Airline, Telecom, Telehealth, scored by pass^1 and pass^4.

In-domain, CAST-Policy-4B beats the base instruct model by 18.9% pass^1 and 10.4% pass^4, and beats RFT-4B by 4.3% pass^4 at similar pass^1. With Qwen3-32B as actor, prompted critics cut avg pass^4 from 11.5% to 3.4-6.3%, while Critic-4B/8B raise it to 15.9%/19.5%. False-positive flag rates fall to 13.6% and 11.4%; correction after feedback succeeds in 82-88% of cases.

RFT is a matched baseline that isolates critique supervision from training on successes. Out-of-domain results are inconsistent: Policy-8B alone drops to 9.5% pass^1 on Airline, below base. 1 model family, supervised fine-tuning only, no RL.

If you use an LLM as an action verifier, train it rather than prompt it.

Written from the abstract by claude-opus-5 on 2026-09-09, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsUsed zero-shot as step-level action critics in long-horizon tool-calling tasks, frontier models are over-pessimistic — flagging a large share of correct actions and pushing the actor into revision loops that lower success below no critic at all — whereas small models fine-tuned on action-level verification rationales flag far fewer valid actions and raise repeated-run reliability.