single-step-tool-calling-agents-near-perfect-clean-accuracy-does-predict-robustness CLM-0638For single-step tool-calling agents, near-perfect clean accuracy does not predict robustness, and the dominant weak point is recovering the answer from perturbed tool returns rather than selecting the tool or binding arguments: across seven models, clean success averaged 0.979 while success under corrupted or missing returned evidence averaged 0.455, with no model above 0.60.
Capability: Using the tools it is given
Observed on
Single-step tool calls over 16 deterministic local tools with fixed gold arguments; four stage-aligned perturbation families at three severities; no long-horizon or stateful depend.
Sources
- supportsToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling AgentsMeasured on 15,456 single-family records across 7 models, 14 subtypes; family-mean robustness 0.918 interface, 0.773 intent, 0.688 runtime, 0.455 observation. A human audit found 68.3% of sampled evidence-loss cases at least partially recoverable, so the gap is not an unsolvable-target artifact. Deterministic local tools, so absolute rates may not transfer to live APIs.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For single-step tool-calling agents, near-ceiling accuracy on unperturbed calls does not predict robustness to perturbation, and corrupting or removing the tool's returned evidence hurts far more than perturbing tool names, descriptions, or the user request.Using the tools it is given · reviewed by AI
- When a tool call is forced and returns an unusable payload, tool-augmented agents report the failure honestly almost always if the response envelope declares an error, but assert unsupported values or invent policy reasons in a large share of cases when the envelope says status:ok while withholding the value (redacted, corrupted, stale, empty, malformed, truncated).Stating false facts confidently · reviewed by AI
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- For multi-turn tool-using service agents, counting excess user-facing turns above a task-specific human reference predicts human efficiency preferences between two successful trajectories (about 91% accuracy on turn-differing pairs), while counting excess tool calls does not (chance on pure-tool pairs), because a raw call count cannot say whether a call verified a constraint or wasted effort.Using the tools it is given · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A model-family evaluation under the same stage-aligned perturbations where observation/evidence perturbation is no more damaging than interface or intent perturbation, or where clean accuracy predicts per-family robustness ranking. Drafted stance toward single-step-tool-calling-agents-near-ceiling-accuracy-unperturbed-calls-does-predict: supports -- Independently reproduces that corrupting the tool's returned evidence hurts far more than perturbing tool names, descriptions, or the user request.