tool-use-fine-tuning-closes-a-gap-rather-than-extending-a-frontierobservationsingle paper
The large reported gains from fine-tuning on tool-call trajectories come from small open models starting far behind — distilling a few hundred trajectories from a stronger model moves them a long way. Whether the same procedure helps a model that is already good at tool use is a separate question this evidence does not answer.
Capability: Using the tools it is given · Agentic, Coding agent
Observed on
2023, Llama2-7B fine-tuned on GPT-4 trajectories.
Sources
- Fine-tuning Llama2-7B on 500 GPT-4-generated agent trajectories gave a 77% HotpotQA improvement, and mixing trajectories from multiple tasks and prompting methods improved agents further.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- With weights frozen and no stronger model in the loop, an agent can raise its own pass rate by rewriting the scaffolding it runs inside — clustering its failures, proposing minimal edits to prompts, tools and policy, and keeping only edits that improve one split without degrading another: all 9 of 9 model-by-benchmark pairs improved, by up to 40.6 percentage points (GLM-5 on AppWorld, 44.4% to 85.0%), with weak models gaining most; but the promotion gate reads the held-out split, so the reported held-out gain is not clean generalisation, and there is no matched-compute baseline.Fixing its own mistakes · unreviewed
- Fine-tuning on a large corpus of real API-call traces markedly improves multi-tool task completion in open models.Using the tools it is given
- For locally deployed open-weight models driving a stateful, dependency-ordered MCP tool server, cutting tool descriptions from full specifications (purpose, parameter semantics, constraints, failure conditions) down to one sentence each raises the fraction of calls the server rejects for every model tested, while its effect on task coverage is less consistent.Using the tools it is given · unreviewed
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · unreviewed
- On the best current calibration, AI is not yet accelerating its own development in a self-sustaining way: the modelled threshold is that a one-unit gain in model capability must buy at least 15% higher AI R&D productivity, and the back-of-envelope figure from reported engineer uplift since coding agents launched is about 9% — below it, but rising, so the gap is a current reading rather than a ceiling.Whether the measurement made the finding
Notes
Deliberately stated as a scope limit rather than as "gains are largest for weak models," which would be a comparative the source does not establish — it reports a large gain for a weak model, which is not the same as measuring the gain shrinking as the baseline rises. If a paper ablates starting strength directly, this claim should be tightened or replaced.