MearcAll papers

arXiv 2302.04761 · cs.CL

Toolformer: Language Models Can Teach Themselves to Use Tools

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, and 2 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_5d3c4f36
In this paper, we show that LMs can teach themselves touse external tools via simple APIs and achieve the best of both worlds.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bab1f613
We conduct experiments on a variety of different downstream tasks, demonstrating that after learning to use tools, Toolformer, which is based on a pretrained GPT-J model (Wang and Komatsuzaki, 2021) … (excerpt)

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6fcfcd84
Beyond that, we also ensure that our approach does not hurt the model’s core language modeling abilities; we verify this by looking at perplexity on two language modeling datasets (Section … (excerpt)

Experiments · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2fbbdf56
We use standard greedy decoding, but with one modification for Toolformer: We let the model start an API call not just when <API> is the most likely 5This is achieved … (excerpt)

Downstream Tasks · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0185b898
Crucially, Toolformer clearly outperforms these baseline models, improving upon the best baseline by 11.7, 5.2 and 18.6 points, respectively.

LAMA · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fd25cf24
It also clearly outperforms OPT (66B) and GPT-3 (175B), despite both models being much larger.

LAMA · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a8c826fc
This is achieved because the model independently decides to ask the question answering tool for the required information in almost all cases (98.1%); for only very few examples, it uses … (excerpt)

LAMA · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9621e685
Toolformer uses the question answering tool for most examples, clearly outperforming all baselines of the same size and achieving results competitive with GPT-3 (175B).

Math Datasets · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7b6739da
Toolformer makes use of the calculator tool for most examples, clearly outperforming even OPT (66B) and GPT-3 (175B). number predicted by the model.7 Table 4 shows results for all benchmarks.

Math Datasets · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f44651c3
Nonetheless, allowing the model to make API calls more than doubles performance for all tasks, and also clearly outperforms the much larger OPT and GPT-3 models.

Math Datasets · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fe24014e
This is because across all benchmarks, for 97.9% of all examples the model decides to ask the calculator tool for help.

Math Datasets · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e2579b7a
Using the Wikipedia search tool for most examples, Toolformer clearly outperforms baselines of the same size, but falls short of GPT-3 (175B). this would make solving the tasks trivial, especially … (excerpt)

Question Answering · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9a1690c6
Once again, Toolformer clearly outperforms all other models based on GPT-J, this time mostly relying on the Wikipedia search API (99.3%) to find relevant information.

Question Answering · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_fc459fec
While using the machine translation tool to translate questions is helpful across all languages, further pretraining on CCNet deteriorates performance; consequently, Toolformer does not consistently outperform GPT-J.

Multilingual Question Answering · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_2dda5da1
The final two rows correspond to models that are given contexts and questions in English. former does not consistently outperform vanilla GPT-J.

Multilingual Question Answering · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_88aad257
Toolformer outperforms all baselines, but does not make use of the calendar tool for TEMP LAMA.

Temporal Datasets · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0faefb80
Results shown in Table 7 illustrate that Toolformer outperforms all baselines for both TEM- PLAMA and DATESET .

Temporal Datasets · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3b004b5b
However, closer inspection shows that improvements on TEMP LAMA can not be attributed to the calendar tool, which is only used for 0.2% of all examples, but mostly to the … (excerpt)

Temporal Datasets · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4a56b917
For DATESET , on the other hand, the considerable improvement of Toolformer compared to other models can be fully accredited to the calendar tool, which it makes use of for … (excerpt)

Temporal Datasets · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c30d8ad3
Figure 4 shows that the ability to leverage the provided tools only emerges at around 775M parameters: smaller models achieve similar performance both with and without tools.

Scaling Laws · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a134d533
This can be seen from the fact that performance on examples where it decides not to make an API call (44.3 and 19.9) is higher than average performance if no … (excerpt)

Analysis · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_27b3dde0
Toolformer considerably improves zero-shot performance of a 6.7B parameter GPT-J model, enabling it to even outperform a much larger GPT-3 model on a range of different downstream tasks.

Conclusion · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2302.04761) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.