multi-turn-tool-calling-benchmarks-score-whether-final-environment-state-matchesOn multi-turn tool-calling benchmarks that score whether the final environment state matches a gold trajectory, aggregate accuracy hides wrong action-class choices: a model can call a withheld or under-specified tool with fabricated arguments, be handed the missing piece on a later turn, and still be graded PASS — so heavily tool-trained model families rank high while almost never asking for a missing parameter or refusing an unavailable function.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Whether the measurement made the finding
Observed on
Multi-turn, state-graded tool benchmarks (BFCL v3 multi-turn, tau2-bench retail/airline) where the simulator supplies missing information on later turns; needs a per-category gold .
Sources
- 26 checkpoints (11 open-weight families, 4 closed anchors) on BFCL v3 multi-turn, 200 conversations per category, cross-checked on tau2-bench; xLAM-2-8b reported at 71.8% aggregate accuracy with 10% miss_func gold-action recall, and a 200-case audit (kappa 0.92) found 92.3% of its miss_func misses invoked the held-out function. Design shows the grader is blind to action class; it does not establish which training stage causes the miscalibration, and gold action classes come from benchmark design rather than a deployable detector.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · unreviewed
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · unreviewed
- When a calculator-using model is trained with reinforcement learning on verifiable final-answer rewards for an arithmetic search task (Countdown), the gain lands almost entirely at low k — pass@1 rises sharply while pass@16 moves little or falls — because the update can only reinforce correct trajectories the starting policy already samples, and prompt groups with no correct sample supply no gradient.Using the tools it is given · unreviewed
- When research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.Whether the measurement made the finding · unreviewed
- When training small instruction-tuned models with reinforcement learning plus factual supervision, routing each atomic fact's verification score only to the tokens that produced it, and down-weighting verifier judgements that do not change when their key evidence is removed, improves factuality benchmark scores over trajectory-level or reasoning- step-level factual rewards.Checking claims against evidence · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Compute per-turn gold action class agreement on the same benchmarks and find it tracks state-graded accuracy — no cases where accuracy exceeds gold-action recall, and no family with high aggregate score and low ask/refuse emission. Proposed technique, not catalogued: Gold Action Recall alongside state-graded accuracy.