single-step-tool-calling-agents-near-ceiling-accuracy-unperturbed-calls-does-predict CLM-0487For single-step tool-calling agents, near-ceiling accuracy on unperturbed calls does not predict robustness to perturbation, and corrupting or removing the tool's returned evidence hurts far more than perturbing tool names, descriptions, or the user request.
Capability: Using the tools it is given
Observed on
Seven frontier models on a deterministic local 16-tool environment, single-step calls, four perturbation families at light/medium/heavy severity; no long-horizon or live-API settin.
Sources
- supportsToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling AgentsValidated by: Counterfactual repair at the attributed stage resolved 71/98 failures (100% single-error, 82.1% cascade, 0% multi-fault); human audit of 30 ambiguous cases agreed 74.2%; labels predicted that an S1 alias fix raised gpt-5.4-mini from 21/30 to 30/30 while an S3 CoT fix left it unchanged at 21/30.. Measured over 15,456 single-family records across 7 models: mean clean 0.979 versus overall robustness 0.664-0.766; family means 0.918 interface, 0.773 intent, 0.688 runtime, 0.455 observation, with no model above 0.60 on observation. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For single-step tool-calling agents, near-perfect clean accuracy does not predict robustness, and the dominant weak point is recovering the answer from perturbed tool returns rather than selecting the tool or binding arguments: across seven models, clean success averaged 0.979 while success under corrupted or missing returned evidence averaged 0.455, with no model above 0.60.Using the tools it is given · reviewed by AI
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- For multi-turn tool-using service agents, counting excess user-facing turns above a task-specific human reference predicts human efficiency preferences between two successful trajectories (about 91% accuracy on turn-differing pairs), while counting excess tool calls does not (chance on pure-tool pairs), because a raw call count cannot say whether a call verified a constraint or wasted effort.Using the tools it is given · reviewed by AI
- On cross-system urban tasks where several services must be chained under matched tools and budgets, the task-success gap between orchestration strategies is not explained by required-tool coverage, call ordering, or asking for missing inputs — those are near-saturated across strategies — but by whether the agent goes beyond the minimal toolchain to refine coarse results and verify each candidate against the stated constraints before answering.Using the tools it is given · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A benchmark where models' clean tool-call accuracy tracks their accuracy under perturbation, or where returned-evidence corruption costs less accuracy than interface/intent perturbation of matched severity.