open-models-8b-14b-filling-structured-tool-call-parameters-linear-probe CLM-0624For open models (8B-14B) filling structured tool-call parameters, a linear probe on the hidden state at the position just before a parameter value is written predicts whether that value will be correct better than token log-probability (in-domain AUC about 0.986 vs 0.914), and using that score to filter self-generated calls for fine-tuning beats self-training filtered by log-probability or majority vote.
Capability: Using the tools it is given
Observed on
Probe must be fit per dataset, model and sampling temperature; it degrades outside that setting. Benchmarks adapted so tool selection is removed and only parameter filling is score.
Sources
- Validated by: Exact match and field-level F1 across 5 open models and 7 benchmarks: average EM 19.7 (0-shot) to 51.6 (seed SFT) to 59.6 (probe-filtered), above self-training (53.9), log-probability filtering (44.2) and majority vote (48.9); probe-filtered beat seed SFT in all 35 model-dataset pairs; reranking added 4.2-4.6 EM.. Measured on 5 open base models across ParamBench plus 6 adapted external benchmarks, with matched-count filtering baselines. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- For agentic tool calls where a parameter is fabricated rather than grounded in the dialogue, sampling-based uncertainty detectors (token entropy, semantic entropy, SEP) fail at near-chance because the model fabricates plausible parameters with high confidence, while the model's hidden states at the moment the parameter is generated do carry a separable grounding signal.Stating false facts confidently · reviewed by AI
- For long-horizon LLM agents on tool-use and terminal benchmarks, token-level confidence (predictive entropy, sequence probability) barely separates successful from failed trajectories, and the separable signal lives in the dependency structure of the trajectory: scoring a trajectory by propagating per-step uncertainty over a graph of reasoning, tool-call and environment-feedback nodes raises failure-detection AUROC over both local-confidence and sequence-based agent UQ baselines.Using the tools it is given · reviewed by AI
- On multi-turn tool-calling benchmarks that score whether the final environment state matches a gold trajectory, aggregate accuracy hides wrong action-class choices: a model can call a withheld or under-specified tool with fabricated arguments, be handed the missing piece on a later turn, and still be graded PASS — so heavily tool-trained model families rank high while almost never asking for a missing parameter or refusing an unavailable function.Whether the measurement made the finding · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A probe on pre-emission hidden states that does not beat log-probability at predicting parameter correctness, or probe-filtered self-training that fails to beat log-probability/majority-vote filtering at matched data volume. Drafted stance toward linear-probe-tool-calling-models-hidden-state-last-generated-token: supports -- Both find hidden-state probes detect incorrect tool-call arguments that schema checks pass; this paper moves the read point to before each parameter value and acts on the signal.