clm_5d3c4f36In this paper, we show that LMs can teach themselves touse external tools via simple APIs and achieve the best of both worlds.
arXiv 2302.04761 · cs.CL
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_5d3c4f36In this paper, we show that LMs can teach themselves touse external tools via simple APIs and achieve the best of both worlds.
res_bab1f613We conduct experiments on a variety of different downstream tasks, demonstrating that after learning to use tools, Toolformer, which is based on a pretrained GPT-J model (Wang and Komatsuzaki, 2021) … (excerpt)
res_6fcfcd84Beyond that, we also ensure that our approach does not hurt the model’s core language modeling abilities; we verify this by looking at perplexity on two language modeling datasets (Section … (excerpt)
res_2fbbdf56We use standard greedy decoding, but with one modification for Toolformer: We let the model start an API call not just when <API> is the most likely 5This is achieved … (excerpt)
res_0185b898Crucially, Toolformer clearly outperforms these baseline models, improving upon the best baseline by 11.7, 5.2 and 18.6 points, respectively.
res_fd25cf24It also clearly outperforms OPT (66B) and GPT-3 (175B), despite both models being much larger.
res_a8c826fcThis is achieved because the model independently decides to ask the question answering tool for the required information in almost all cases (98.1%); for only very few examples, it uses … (excerpt)
res_9621e685Toolformer uses the question answering tool for most examples, clearly outperforming all baselines of the same size and achieving results competitive with GPT-3 (175B).
res_7b6739daToolformer makes use of the calculator tool for most examples, clearly outperforming even OPT (66B) and GPT-3 (175B). number predicted by the model.7 Table 4 shows results for all benchmarks.
res_f44651c3Nonetheless, allowing the model to make API calls more than doubles performance for all tasks, and also clearly outperforms the much larger OPT and GPT-3 models.
res_fe24014eThis is because across all benchmarks, for 97.9% of all examples the model decides to ask the calculator tool for help.
res_e2579b7aUsing the Wikipedia search tool for most examples, Toolformer clearly outperforms baselines of the same size, but falls short of GPT-3 (175B). this would make solving the tasks trivial, especially … (excerpt)
res_9a1690c6Once again, Toolformer clearly outperforms all other models based on GPT-J, this time mostly relying on the Wikipedia search API (99.3%) to find relevant information.
clm_fc459fecWhile using the machine translation tool to translate questions is helpful across all languages, further pretraining on CCNet deteriorates performance; consequently, Toolformer does not consistently outperform GPT-J.
clm_2dda5da1The final two rows correspond to models that are given contexts and questions in English. former does not consistently outperform vanilla GPT-J.
clm_88aad257Toolformer outperforms all baselines, but does not make use of the calendar tool for TEMP LAMA.
res_0faefb80Results shown in Table 7 illustrate that Toolformer outperforms all baselines for both TEM- PLAMA and DATESET .
res_3b004b5bHowever, closer inspection shows that improvements on TEMP LAMA can not be attributed to the calendar tool, which is only used for 0.2% of all examples, but mostly to the … (excerpt)
res_4a56b917For DATESET , on the other hand, the considerable improvement of Toolformer compared to other models can be fully accredited to the calendar tool, which it makes use of for … (excerpt)
res_c30d8ad3Figure 4 shows that the ability to leverage the provided tools only emerges at around 775M parameters: smaller models achieve similar performance both with and without tools.
res_a134d533This can be seen from the fact that performance on examples where it decides not to make an API call (44.3 and 19.9) is higher than average performance if no … (excerpt)
res_27b3dde0Toolformer considerably improves zero-shot performance of a 6.7B parameter GPT-J model, enabling it to even outperform a much larger GPT-3 model on a range of different downstream tasks.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2302.04761) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.