tool-augmented-agents-emit-calls-over-unconstrained-raw-json-surface-rather CLM-0495When tool-augmented agents emit calls over an unconstrained raw-JSON surface rather than a schema-enforcing function-calling API, every tested hosted model fabricates tool names or undeclared arguments at rates of roughly 13-57%, and parameter count does not reduce this — an 8B and a 675B model hallucinate at comparable raw-JSON rates.
Capability: Stating false facts confidently
Observed on
Ten hosted Bedrock models (Claude, Nova, GPT-OSS, Llama-3.1, Mistral families), 60 adversarial probes per model per surface, a synthetic 100-tool registry with 10 tools exposed; lo.
Sources
- Validated by: On 322 live-model hallucinations and 400 scripted trials per class, the resolver rejected all H1-H3 cases while a gating-only stack executed them; but both the 322/322 execution and the 0/322 rejection are by-construction consequences of the fail-open threat model, and zero honest over-rejection is measured on the resolver's own honest-call generator, so only the emission counts are independent evidence.. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- For single-step tool-calling agents, near-ceiling accuracy on unperturbed calls does not predict robustness to perturbation, and corrupting or removing the tool's returned evidence hurts far more than perturbing tool names, descriptions, or the user request.Using the tools it is given · reviewed by AI
- For locally deployed open-weight models driving a stateful, dependency-ordered MCP tool server, cutting tool descriptions from full specifications (purpose, parameter semantics, constraints, failure conditions) down to one sentence each raises the fraction of calls the server rejects for every model tested, while its effect on task coverage is less consistent.Using the tools it is given · reviewed by AI
- When an agent must work over a very large tool catalogue, wrapping each tool in a natural-language interface that resolves the API schema internally — so raw schemas are retrieved on demand rather than enumerated in the model's prompt — contributes more to multi-step tool- call success than the surrounding planner, router, or verifier stages.Using the tools it is given · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Running the same adversarial probes and finding fabricated-tool and undeclared-argument rates on a raw-JSON surface fall sharply with model scale within a family, or that raw-JSON and schema-enforced surfaces produce comparable emission rates once refusals are accounted for.