multi-turn-tool-calling-benchmarks-score-whether-final-environment-state-matches CLM-0103On multi-turn tool-calling benchmarks that score whether the final environment state matches a gold trajectory, aggregate accuracy hides wrong action-class choices: a model can call a withheld or under-specified tool with fabricated arguments, be handed the missing piece on a later turn, and still be graded PASS — so heavily tool-trained model families rank high while almost never asking for a missing parameter or refusing an unavailable function.
Capability: Whether the measurement made the finding
Observed on
Multi-turn, state-graded tool benchmarks (BFCL v3 multi-turn, tau2-bench retail/airline) where the simulator supplies missing information on later turns; needs a per-category gold .
Sources
- 26 checkpoints (11 open-weight families, 4 closed anchors) on BFCL v3 multi-turn, 200 conversations per category, cross-checked on tau2-bench; xLAM-2-8b reported at 71.8% aggregate accuracy with 10% miss_func gold-action recall, and a 200-case audit (kappa 0.92) found 92.3% of its miss_func misses invoked the held-out function. Design shows the grader is blind to action class; it does not establish which training stage causes the miscalibration, and gold action classes come from benchmark design rather than a deployable detector.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- When models must apply abstract, entity-independent procedural rules drawn from a shared pool, accuracy is limited by locating the right rules rather than by applying them: giving only the gold rules yields much higher exact-match than embedding those same rules in a pool of 100 or 1,000, and accuracy falls as the pool grows.Following an unfamiliar procedure · reviewed by AI
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- For single-step tool-calling agents, near-ceiling accuracy on unperturbed calls does not predict robustness to perturbation, and corrupting or removing the tool's returned evidence hurts far more than perturbing tool names, descriptions, or the user request.Using the tools it is given · reviewed by AI
- For multi-turn tool-using service agents, counting excess user-facing turns above a task-specific human reference predicts human efficiency preferences between two successful trajectories (about 91% accuracy on turn-differing pairs), while counting excess tool calls does not (chance on pure-tool pairs), because a raw call count cannot say whether a call verified a constraint or wasted effort.Using the tools it is given · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Compute per-turn gold action class agreement on the same benchmarks and find it tracks state-graded accuracy — no cases where accuracy exceeds gold-action recall, and no family with high aggregate score and low ask/refuse emission. Proposed technique, not catalogued: Gold Action Recall alongside state-graded accuracy.