cross-system-urban-tasks-where-several-services-must-be-chained CLM-0557On cross-system urban tasks where several services must be chained under matched tools and budgets, the task-success gap between orchestration strategies is not explained by required-tool coverage, call ordering, or asking for missing inputs — those are near-saturated across strategies — but by whether the agent goes beyond the minimal toolchain to refine coarse results and verify each candidate against the stated constraints before answering.
Capability: Using the tools it is given
Observed on
250-query UrbanEval benchmark over MCP urban services; main controlled setting GPT-5-mini, replicated in rank only across Gemini-2.5-flash, DeepSeek-V4-flash, Qwen3-235B-A22B; LLM .
Sources
- Validated by: Task success rate on UrbanEval: 0.71 overall vs 0.61 for the best baselines (native tool-calling, AutoGen); 64.0% vs 53.5% on the 200 executable queries; incomplete-task failures 20 vs 31-45; valid proactive augmentation rate 0.96 vs 0.84; cost about 3x tokens per success.. Measured on 250 queries, four backbones, four orchestration baselines with matched tools, schemas, temperature and budget. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For multi-turn tool-using service agents, counting excess user-facing turns above a task-specific human reference predicts human efficiency preferences between two successful trajectories (about 91% accuracy on turn-differing pairs), while counting excess tool calls does not (chance on pure-tool pairs), because a raw call count cannot say whether a call verified a constraint or wasted effort.Using the tools it is given · reviewed by AI
- For single-step tool-calling agents, near-ceiling accuracy on unperturbed calls does not predict robustness to perturbation, and corrupting or removing the tool's returned evidence hurts far more than perturbing tool names, descriptions, or the user request.Using the tools it is given · reviewed by AI
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- For locally deployed open-weight models driving a stateful, dependency-ordered MCP tool server, cutting tool descriptions from full specifications (purpose, parameter semantics, constraints, failure conditions) down to one sentence each raises the fraction of calls the server rejects for every model tested, while its effect on task coverage is less consistent.Using the tools it is given · reviewed by AI
- For LLM agents that must compose and then execute domain tool workflows, training separate adapters for the two roles splits the gains cleanly: supervised fine-tuning of the planner on reference tool sequences mainly improves workflow-order fidelity, while node-level RL on argument-match rewards mainly improves tool-argument grounding and final-answer quality, and combining both beats either on a 9B backbone.Planning a sequence of actions before acting · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A controlled comparison on the same benchmark showing that per-strategy differences in tool coverage, dependency ordering, or clarification rate predict the success gap, or that adding candidate verification and augmentation to a baseline leaves success unchanged.