sub-2b-open-weight-models-emitting-json-tool-calls-prompts-already CLM-0221For sub-2B open-weight models emitting JSON tool calls from prompts that already name the tool and all argument values, reported success depends almost entirely on a post-hoc recovery layer that strips code fences and extracts a brace-delimited substring: across 1,000 generations only 5 whole raw responses parsed as JSON directly, while the best condition reached 79% after recovery.
Capability: Using the tools it is given
Observed on
Five checkpoints below 2B (Phi-1.5, Pythia-1.4B, TinyLlama-1.1B-Chat, Qwen2.5-0.5B/1.5B), no chat templates applied, 100 fully specified single-call prompts, no tool execution, gre.
Sources
- Single small benchmark: 100 templated prompts per condition, 10 conditions, Wilson intervals and McNemar tests reported. Base models were run without chat templates and generation was capped at 150 tokens, so truncation and template mismatch are plausible contributors to the low strict-JSON rate; the design does not isolate them.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When tool-augmented agents emit calls over an unconstrained raw-JSON surface rather than a schema-enforcing function-calling API, every tested hosted model fabricates tool names or undeclared arguments at rates of roughly 13-57%, and parameter count does not reduce this — an 8B and a 675B model hallucinate at comparable raw-JSON rates.Stating false facts confidently · reviewed by AI
- For response-level hallucination detection in retrieval-augmented settings, having the model first convert the reference documents into a relational SQL database and then check the response against SELECT query results raises precision and macro F1 over zero-shot direct prediction, over atomic-claim decomposition, and over flat JSON key-value extraction, without any task-specific fine-tuning.Stating false facts confidently · reviewed by AI
- For locally deployed open-weight models driving a stateful, dependency-ordered MCP tool server, cutting tool descriptions from full specifications (purpose, parameter semantics, constraints, failure conditions) down to one sentence each raises the fraction of calls the server rejects for every model tested, while its effect on task coverage is less consistent.Using the tools it is given · reviewed by AI
- When extracting structured rules from multimodal enterprise guideline documents, wrapping the extraction in deterministic schema validation, an LLM semantic judge, and a duplicate/contradiction consistency filter over a shared rule store lowers the share of extracted rules with no matching source sentence from about 15.7% to 3.2% relative to a one-pass pipeline using the same parser and models.Stating false facts confidently · reviewed by AI
- In 7-8B open-weight instruct models (Llama-3.1, Mistral-v0.3, Qwen2.5), the drop in a token's probability when a contiguous mid-layer MLP span is zero-ablated is near zero for function words, elevated for knowledge-bearing tokens, and abnormally spiked for hallucinated tokens, so this internal attribution signal — approximable by a small probe from one forward pass — can be used in beam scoring to raise factuality on open-ended and long-form generation, while leaving strict short-answer EM/F1 unchanged or worse.Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Re-running comparable sub-2B checkpoints (with their official chat templates) and finding that most raw responses are directly parseable as JSON, so recovery adds little to measured success.