multi-turn-tool-using-service-agents-counting-excess-user-facing-turns-above CLM-0454For multi-turn tool-using service agents, counting excess user-facing turns above a task-specific human reference predicts human efficiency preferences between two successful trajectories (about 91% accuracy on turn-differing pairs), while counting excess tool calls does not (chance on pure-tool pairs), because a raw call count cannot say whether a call verified a constraint or wasted effort.
Capability: Using the tools it is given
Observed on
58 Chinese-language ridehailing tasks, 24 non-thinking models, 320 calibration and 80 held-out blinded pairwise preference votes; single domain and language, mostly one primary ann.
Sources
- Validated by: Held-out task-disjoint pairwise preference prediction: 78.7% accuracy, Kendall tau_b 0.573, versus 0.773 for equal-cost length, 0.394 tool-only, 0.925 turn-only at 0.71 coverage; turn-first lexicographic tied at 0.800; plus a 10-task listwise check (all tau_b positive, mean 0.488) and stability to +/-1-2 floor perturbations.. Measured on 75 directional held-out pairs: 90.6% accuracy on 53 turn-differing pairs but only 58.3% on the 12 with small turn gaps, and 50.0% on 22 pure-tool pairs with an exactly even 11/11 human vote split. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For single-step tool-calling agents, near-ceiling accuracy on unperturbed calls does not predict robustness to perturbation, and corrupting or removing the tool's returned evidence hurts far more than perturbing tool names, descriptions, or the user request.Using the tools it is given · reviewed by AI
- On cross-system urban tasks where several services must be chained under matched tools and budgets, the task-success gap between orchestration strategies is not explained by required-tool coverage, call ordering, or asking for missing inputs — those are near-saturated across strategies — but by whether the agent goes beyond the minimal toolchain to refine coarse results and verify each candidate against the stated constraints before answering.Using the tools it is given · reviewed by AI
- On multi-turn tool-calling benchmarks that score whether the final environment state matches a gold trajectory, aggregate accuracy hides wrong action-class choices: a model can call a withheld or under-specified tool with fabricated arguments, be handed the missing piece on a later turn, and still be graded PASS — so heavily tool-trained model families rank high while almost never asking for a missing parameter or refusing an unavailable function.Whether the measurement made the finding · reviewed by AI
- For single-step tool-calling agents, near-perfect clean accuracy does not predict robustness, and the dominant weak point is recovering the answer from perturbed tool returns rather than selecting the tool or binding arguments: across seven models, clean success averaged 0.979 while success under corrupted or missing returned evidence averaged 0.455, with no model above 0.60.Using the tools it is given · reviewed by AI
- When a tool-using agent is trained with outcome-only RL and a surface prompt cue co-occurs with one tool in most training examples, the agent invokes that tool on tasks where it is useless, but only for tools it has already learned to use well and mainly when the cue is semantically related to the tool.Using the tools it is given · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A held-out preference set in which excess tool-call counts alone predict human efficiency judgements well above chance, or in which excess turns fail to predict them, would break the claim.