Toolformer: Language Models Can Teach Themselves to Use Tools

A language model can annotate its own pretraining corpus with API calls and keep only those that reduce loss on the following tokens, and that alone is enough to teach it when to call a tool. Toolformer applies this to GPT-J (6.7B) on a CCNet subset, with 5 tools: a QA system (Atlas), BM25 Wikipedia search, a calculator, a calendar, and NLLB translation. Evaluation is zero-shot, no in-context examples.

On LAMA subsets it beats the best same-size baseline by 11.7, 5.2 and 18.6 points and passes GPT-3 (175B); on math it reaches 40.4 / 29.4 / 44.0 on ASDiv / SVAMP / MAWPS against GPT-3's 14.0 / 10.0 / 19.8, calling the calculator on 97.9% of examples. On Dateset it gets 27.3 vs 3.9 for GPT-J.

The design isolates the effect reasonably: baselines include GPT-J finetuned on the same text without API calls, and Toolformer with API calls disabled at decoding. Perplexity is unchanged (10.3 WikiText, 10.5 CCNet) with calls off.

Boundaries: it lags GPT-3 on WebQS/NQ/TriviaQA, does not consistently beat GPT-J on multilingual MLQA, ignores the calendar on TempLAMA (0.2% use), cannot chain or refine calls, and the ability only emerges around 775M parameters. 1 model family, 1 corpus.

Useful if you are deciding whether tool use needs human demonstrations.