MearcAll papers

arXiv 2109.01652 · cs.CL

Finetuned Language Models Are Zero-Shot Learners

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, and 3 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_8ee256f4
We show that instruction tuning—finetuning language models on a collection of datasets described via instructions—substantially improves zeroshot performance on unseen tasks.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b2bfedd3
FLAN substantially improves the performance of its unmodified counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 datasets that we evaluate.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8a633570
FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a2bc3546
Bottom: performance of zero-shot FLAN, compared with zero-shot and few-shot GPT-3, on three unseen task types where instruction tuning improved performance substantially out of ten we evaluate.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a52a21d0
Our evaluations show that FLAN substantially improves the zero-shot performance of the base 137B-parameter model.

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e0d58b31
FLAN’s zero-shot also outperforms 175B-parameter GPT-3’s zero-shot on 20 of 25 datasets that we evaluate, and even outperforms GPT-3’s few-shot by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, … (excerpt)

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2352beec
Instruction tuning is a simple method that, as depicted in Figure 2, combines appealing aspects of both the pretrain–finetune and prompting paradigms by using supervision via finetuning to improve language … (excerpt)

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_19bde5e8
With the best dev template, zero-shot FLAN outperforms zero-shot GPT-3 on 20 of 25 datasets and even surpasses GPT-3’s few-shot performance on 10 datasets.

R ESULTS · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f2f6a0a9
With the best dev-template, zero-shot FLAN outperforms zero-shot GLaM on 13 of 19 available datasets and one-shot GLaM on 11 of 19 datasets.

R ESULTS · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_c6a99161
On five NLI datasets, where a model must determine whether a hypothesis is true given some premise, FLAN outperforms all baselines by a large margin.

R ESULTS · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_30edc0cc
For closed-book QA, which asks models to answer questions about the world without access to specific information containing the answer, FLAN outperforms GPT-3 on all four datasets.

R ESULTS · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b86e9ce8
Compared with GPT-3, FLAN outperforms zero-shot GPT-3 for all six evaluations, though it underperforms few-shot GPT-3 in most cases.

R ESULTS · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a93ca9da
For seven commonsense reasoning and coreference resolution tasks (see Table 2 in the Appendix), FLAN only outperforms LaMDA-PT on three of the seven tasks.

R ESULTS · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_1bbd2ae4
Generally, zero-shot FLAN outperforms zero-shot LaMDA-PT and is comparable with or better than few-shot LaMDA-PT.

R ESULTS · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_cf3c9840
As expected, we observe that average performance across the three held-out clusters improves as we add additional clusters and tasks to instruction tuning (with the exception of the sentiment analysis … (excerpt)

N UMBER OF INSTRUCTION TUNING CLUSTERS · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_867502a4
We see that for the two models on the order of 100B parameters, instruction tuning substantially improves performance on held-out tasks, as is expected given the prior results in our … (excerpt)

S CALING LAWS · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a14e2bfb
As shown in Figure 9, few-shot exemplars improve the performance on all task clusters, compared with zero-shot FLAN.

I NSTRUCTIONS WITH FEW-SHOT EXEMPLARS · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_56f6f4a8
Our prompt tuning setup follows the procedure of Lester et al. (2021) except that we use a prompt length of 10, weight decay of 1e-4, and did not use dropout … (excerpt)

I NSTRUCTION TUNING FACILITATES PROMPT TUNING · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_32604032
In many cases, especially for the low-resource setting, prompt tuning on FLAN even achieves more than 10% improvement over prompt tuning on the LaMDA-PT.

I NSTRUCTION TUNING FACILITATES PROMPT TUNING · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_431babf4
Our instruction-tuned model, FLAN, improves performance against an untuned model and surpasses zero-shot GPT-3 on the majority of tasks that we evaluate on.

D ISCUSSION · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ad77d19c
Our primary hypothesis is that instruction tuning on a diverse set of tasks improves performance on unseen tasks. §4.1 showed that adding more task clusters improves performance; here, we further … (excerpt)

D ATASETS PER TASK CLUSTER & T EMPLATES PER DATASET · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_90f9a410
Using more datasets per cluster improved performance by almost 10% on average across the three held-out clusters.

D ATASETS PER TASK CLUSTER & T EMPLATES PER DATASET · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_806c77b7
Notably, training with task/dataset name achieved a high NLI score largely because it achieved a score of 83.9 on the CB dataset, for which the validation set only has 56 … (excerpt)

D ATA CONTAMINATION ANALYSIS · page 25 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b81123d9
As ANLI R1 and R2 are based entirely from Wikipedia examples (R3 is not), we posit that this higher degree of contamination in our pretraining dataset compared with GPT-3’s is … (excerpt)

D ATA CONTAMINATION ANALYSIS · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8265025b
Of the remaining datasets, only ReCoRD and PIQA had a clean subset performance that was lower than the overall evaluation set performance by more than 1%.

D ATA CONTAMINATION ANALYSIS · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1f99115a
As shown by our experiments in §4.5, prompt tuning is an orthogonal method for which instruction tuning can additionally improve performance.

P ROMPTING · page 28 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_dfde925f
Figure 7A confirms that scale improves performance for tasks that were seen during instruction tuning, as expected.

F REQUENTLY ASKED QUESTIONS · page 30 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_dc36669d
30 Published as a conference paper at ICLR 2022 Nonetheless, the fact that scale does not always improve zero-shot performance of untuned LaMDA- PT is an interesting artifact.

F REQUENTLY ASKED QUESTIONS · page 30 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2109.01652) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.