clm_8ee256f4We show that instruction tuning—finetuning language models on a collection of datasets described via instructions—substantially improves zeroshot performance on unseen tasks.
arXiv 2109.01652 · cs.CL
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_8ee256f4We show that instruction tuning—finetuning language models on a collection of datasets described via instructions—substantially improves zeroshot performance on unseen tasks.
res_b2bfedd3FLAN substantially improves the performance of its unmodified counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 datasets that we evaluate.
res_8a633570FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze.
res_a2bc3546Bottom: performance of zero-shot FLAN, compared with zero-shot and few-shot GPT-3, on three unseen task types where instruction tuning improved performance substantially out of ten we evaluate.
res_a52a21d0Our evaluations show that FLAN substantially improves the zero-shot performance of the base 137B-parameter model.
res_e0d58b31FLAN’s zero-shot also outperforms 175B-parameter GPT-3’s zero-shot on 20 of 25 datasets that we evaluate, and even outperforms GPT-3’s few-shot by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, … (excerpt)
res_2352beecInstruction tuning is a simple method that, as depicted in Figure 2, combines appealing aspects of both the pretrain–finetune and prompting paradigms by using supervision via finetuning to improve language … (excerpt)
res_19bde5e8With the best dev template, zero-shot FLAN outperforms zero-shot GPT-3 on 20 of 25 datasets and even surpasses GPT-3’s few-shot performance on 10 datasets.
res_f2f6a0a9With the best dev-template, zero-shot FLAN outperforms zero-shot GLaM on 13 of 19 available datasets and one-shot GLaM on 11 of 19 datasets.
clm_c6a99161On five NLI datasets, where a model must determine whether a hypothesis is true given some premise, FLAN outperforms all baselines by a large margin.
res_30edc0ccFor closed-book QA, which asks models to answer questions about the world without access to specific information containing the answer, FLAN outperforms GPT-3 on all four datasets.
res_b86e9ce8Compared with GPT-3, FLAN outperforms zero-shot GPT-3 for all six evaluations, though it underperforms few-shot GPT-3 in most cases.
res_a93ca9daFor seven commonsense reasoning and coreference resolution tasks (see Table 2 in the Appendix), FLAN only outperforms LaMDA-PT on three of the seven tasks.
clm_1bbd2ae4Generally, zero-shot FLAN outperforms zero-shot LaMDA-PT and is comparable with or better than few-shot LaMDA-PT.
clm_cf3c9840As expected, we observe that average performance across the three held-out clusters improves as we add additional clusters and tasks to instruction tuning (with the exception of the sentiment analysis … (excerpt)
res_867502a4We see that for the two models on the order of 100B parameters, instruction tuning substantially improves performance on held-out tasks, as is expected given the prior results in our … (excerpt)
res_a14e2bfbAs shown in Figure 9, few-shot exemplars improve the performance on all task clusters, compared with zero-shot FLAN.
res_56f6f4a8Our prompt tuning setup follows the procedure of Lester et al. (2021) except that we use a prompt length of 10, weight decay of 1e-4, and did not use dropout … (excerpt)
res_32604032In many cases, especially for the low-resource setting, prompt tuning on FLAN even achieves more than 10% improvement over prompt tuning on the LaMDA-PT.
res_431babf4Our instruction-tuned model, FLAN, improves performance against an untuned model and surpasses zero-shot GPT-3 on the majority of tasks that we evaluate on.
res_ad77d19cOur primary hypothesis is that instruction tuning on a diverse set of tasks improves performance on unseen tasks. §4.1 showed that adding more task clusters improves performance; here, we further … (excerpt)
res_90f9a410Using more datasets per cluster improved performance by almost 10% on average across the three held-out clusters.
res_806c77b7Notably, training with task/dataset name achieved a high NLI score largely because it achieved a score of 83.9 on the CB dataset, for which the validation set only has 56 … (excerpt)
res_b81123d9As ANLI R1 and R2 are based entirely from Wikipedia examples (R3 is not), we posit that this higher degree of contamination in our pretraining dataset compared with GPT-3’s is … (excerpt)
res_8265025bOf the remaining datasets, only ReCoRD and PIQA had a clean subset performance that was lower than the overall evaluation set performance by more than 1%.
res_1f99115aAs shown by our experiments in §4.5, prompt tuning is an orthogonal method for which instruction tuning can additionally improve performance.
res_dfde925fFigure 7A confirms that scale improves performance for tasks that were seen during instruction tuning, as expected.
res_dc36669d30 Published as a conference paper at ICLR 2022 Nonetheless, the fact that scale does not always improve zero-shot performance of untuned LaMDA- PT is an interesting artifact.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2109.01652) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.