MearcAll papers

arXiv 2210.11416 · cs.LG cs.CL

Scaling Instruction-Finetuned Language Models

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, and 29 more

Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.

No reproduction recordedres_30a4d206
Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4dbcc91d
We also publicly release Flan-T5 checkpoints,1 which achieve strong few-shot performance even compared to much larger models, such as PaLM 62B.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_27cf7306
Our experiments show that instruction finetuning does scale well with the number of tasks and the size of the model.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_92a1090c
Our experiments show that whereas prior instruction finetuning methods that do not include chain-of-thought (CoT; Wei et al., 2022b) severely degrade performance on CoT evaluations, adding just nine CoT datasets into the finetuning mixture enables better performance on all evaluations.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_97bb641c
Flan-PaLM outperforms PaLM, achieving new state-of-theart on several benchmarks.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_bbfb24f8
In human rater evaluations, Flan-PaLM substantially outperforms PaLM on a challenging set of open-ended generation questions, suggesting improved usability.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_b7d7f18a
These checkpoints have strong zeroshot, few-shot, and CoT abilities, outperforming prior public checkpoints such as T5 (Raffel et al., 2020).

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b81abd9e
Overall, our results underscore how instruction finetuning can improve performance across a range of models, prompting setups, and evaluation tasks. - Random 25.0 - Average human rater 34.5 May 2020 GPT-3 5-shot 43.9 Mar.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5aedfbfe
5A normalized metric scales an evaluation number with respect to a task-specific lower bound such as random guessing baseline for a multiple choice question.

Evaluation protocol · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d8ebb7f5
For example, if random guessing produces 50% accuracy and the max accuracy of 100%, then a raw accuracy of 55% would be be normalized to 10%, and a raw accuracy of 45% would be normalized to -10% since it is worse than random.

Evaluation protocol · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_70fcbf01
Second, increasing the number of finetuning tasks improves performance, although the majority of the improvement comes from using up to 282 tasks.

Evaluation protocol · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6ecbd3bc
Another explanation is that most of the gains from multi-task instruction finetuning come from the model learning to better express knowledge that it already knows from pretraining, and more than 282 tasks does not help too much.

Evaluation protocol · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1052df1b
Finally,weseethatincreasingmodelscalebyanorderofmagnitude(i.e.,8B →62Bor62B →540B)improves performance substantially for both finetuned and non-finetuned models.

Evaluation protocol · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_854c03be
First, we show that the improved reasoning abilities of Flan-PaLM surprass prior models on several benchmarks.

Finetuning with chain-of-thought annotations · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_e8f5d67c
Finally, we show that CoT finetuning unlocks zero-shot reasoning via “let’s think step-by-step” on challenging BIG-Bench tasks.

Finetuning with chain-of-thought annotations · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1994bed1
Table 4 shows that CoT prompting abilities of Flan-PaLM outperform PaLM on the four held-out evaluation benchmarks.

Finetuning on chain-of-thought improves reasoning on held-out tasks · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7c0f75fc
On the MGSM benchmark of multilingual math problems, Flan-PaLM with CoT + SC improves SOTA significantly, achieving high performance even on under-represented languages, such as 69.6% on Bengali.

Finetuning on chain-of-thought improves reasoning on held-out tasks · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_36364ebc
In comparison, PaLM with CoT + SC only achieves 63.6% with French and 61.2% on German, which are high-resource languages.

Finetuning on chain-of-thought improves reasoning on held-out tasks · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_425fb847
For example, for BBH-algo, which comprises tasks that require symbolic manipulation only (e.g., keeping the order of a shuffled list of objects, sorting a list of words in alphabetical order), Flan-PaLM does not outperform code-davinci-002, even with CoT + SC.

Finetuning on chain-of-thought improves reasoning on held-out tasks · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8df3f550
These results were quite strong for some benchmarks—for example, Flan-T5-XL is only 3B parameters and achieves a MMLU score of 52.4%, surpassing GPT-3 175B’s score of 43.9%.

Putting it all together · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_094121a6
Scaling curves for instruction finetuning.We showed in Section 3 that two key components of instruction finetuning—the size of the model and the number of finetuning tasks—improve performance.

Discussion · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_a4aa661b
Flan-PaLM outperforms PaLM on several Responsible AI benchmarks, particularly benchmarks measuring toxic language harms.

Discussion · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2def5e55
Instruction finetuning is relatively compute-efficient.Although scaling the size of language models has 13 been shown to reliably improve performance, it requires considerable compute.

Discussion · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6129029a
Instruction finetuning improves the performance of models with a relatively small amount of compute—for instance, for PaLM 540B, instruction finetuning requires only 0.2% of the pre-training compute but improves the normalized average across evaluation benchmarks by 9.4%.

Discussion · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ce986f92
Moreover, smaller models that use instruction finetuning can sometimes outperform larger models without it.

Discussion · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3508831b
Flan-PaLM achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU.

Conclusions · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_63f545ad
Additionally, we show that instruction finetuning is compatible with a range of model sizes, architectures, and pre-training objectives.

Conclusions · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_0de0332a
To this end, we publicly release Flan-T5 models that outperform baseline T5 models by a large margin.

Conclusions · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ba961439
24 A.2 Does using CoT prompting for evaluation always improve performance? . . . . . . . . . . . .

Frequently asked questions 24 · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9c459e4d
24 A.3 Does instruction finetuning improve performance more or less for larger models? . . . . . . .

Frequently asked questions 24 · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f6b9cfff
In Table 5, CoT does not improve performance on MMLU, compared with direct prompting.

Frequently asked questions · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e775f01a
Instruction finetuning improves the normalized average score across four datasets for 8B model by 15.5% and the 540B model by 9.4%.

Frequently asked questions · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_eb2a6704
However, the baseline is also higher for the 540B model, so it could also be relevant to compute the relative reduction in error rate, which was 18.4% for the 540B model and 16.6% for the 8B model.

Frequently asked questions · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_565f54c1
Table 7 shows that toxicity classification capabilities increase significantly with instruction finetuning, as we observe a consistent improvement both in zero and few-shot classification.

Ability to classify toxic content · page 29 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6bed31c1
All the instruction finetuned models in a zero-shot classification scenario outperform the base models, even when comparing with their performance in the 10-shot scenario.

Ability to classify toxic content · page 29 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_96dd260d
However, these models are not yet on a par with systems specifically designed for this task, such as Perspective API, which achieves an AUC on 97.0% on the same evaluation dataset.

Ability to classify toxic content · page 29 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_24e0a40b
Interestingly, scale does not have a clear linear influence for Flan-PaLM models and Figure 15 shows Flan-T5-XXL outperforms all Flan-PaLM variations.

Gender and Occupation Bias · page 29 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_d3c51e33
Interestingly, Flan-T5-XXL outperforms Flan- PaLM models and the influence of scale is not clear and linear.

Gender and Occupation Bias · page 31 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2210.11416) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.