res_30a4d206Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU.
arXiv 2210.11416 · cs.LG cs.CL
Read on arXiv · CC BY 4.0
Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.
res_30a4d206Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU.
res_4dbcc91dWe also publicly release Flan-T5 checkpoints,1 which achieve strong few-shot performance even compared to much larger models, such as PaLM 62B.
clm_27cf7306Our experiments show that instruction finetuning does scale well with the number of tasks and the size of the model.
clm_92a1090cOur experiments show that whereas prior instruction finetuning methods that do not include chain-of-thought (CoT; Wei et al., 2022b) severely degrade performance on CoT evaluations, adding just nine CoT datasets into the finetuning mixture enables better performance on all evaluations.
clm_97bb641cFlan-PaLM outperforms PaLM, achieving new state-of-theart on several benchmarks.
clm_bbfb24f8In human rater evaluations, Flan-PaLM substantially outperforms PaLM on a challenging set of open-ended generation questions, suggesting improved usability.
clm_b7d7f18aThese checkpoints have strong zeroshot, few-shot, and CoT abilities, outperforming prior public checkpoints such as T5 (Raffel et al., 2020).
res_b81abd9eOverall, our results underscore how instruction finetuning can improve performance across a range of models, prompting setups, and evaluation tasks. - Random 25.0 - Average human rater 34.5 May 2020 GPT-3 5-shot 43.9 Mar.
res_5aedfbfe5A normalized metric scales an evaluation number with respect to a task-specific lower bound such as random guessing baseline for a multiple choice question.
res_d8ebb7f5For example, if random guessing produces 50% accuracy and the max accuracy of 100%, then a raw accuracy of 55% would be be normalized to 10%, and a raw accuracy of 45% would be normalized to -10% since it is worse than random.
res_70fcbf01Second, increasing the number of finetuning tasks improves performance, although the majority of the improvement comes from using up to 282 tasks.
res_6ecbd3bcAnother explanation is that most of the gains from multi-task instruction finetuning come from the model learning to better express knowledge that it already knows from pretraining, and more than 282 tasks does not help too much.
res_1052df1bFinally,weseethatincreasingmodelscalebyanorderofmagnitude(i.e.,8B →62Bor62B →540B)improves performance substantially for both finetuned and non-finetuned models.
clm_854c03beFirst, we show that the improved reasoning abilities of Flan-PaLM surprass prior models on several benchmarks.
clm_e8f5d67cFinally, we show that CoT finetuning unlocks zero-shot reasoning via “let’s think step-by-step” on challenging BIG-Bench tasks.
res_1994bed1Table 4 shows that CoT prompting abilities of Flan-PaLM outperform PaLM on the four held-out evaluation benchmarks.
res_7c0f75fcOn the MGSM benchmark of multilingual math problems, Flan-PaLM with CoT + SC improves SOTA significantly, achieving high performance even on under-represented languages, such as 69.6% on Bengali.
res_36364ebcIn comparison, PaLM with CoT + SC only achieves 63.6% with French and 61.2% on German, which are high-resource languages.
res_425fb847For example, for BBH-algo, which comprises tasks that require symbolic manipulation only (e.g., keeping the order of a shuffled list of objects, sorting a list of words in alphabetical order), Flan-PaLM does not outperform code-davinci-002, even with CoT + SC.
res_8df3f550These results were quite strong for some benchmarks—for example, Flan-T5-XL is only 3B parameters and achieves a MMLU score of 52.4%, surpassing GPT-3 175B’s score of 43.9%.
res_094121a6Scaling curves for instruction finetuning.We showed in Section 3 that two key components of instruction finetuning—the size of the model and the number of finetuning tasks—improve performance.
clm_a4aa661bFlan-PaLM outperforms PaLM on several Responsible AI benchmarks, particularly benchmarks measuring toxic language harms.
res_2def5e55Instruction finetuning is relatively compute-efficient.Although scaling the size of language models has 13 been shown to reliably improve performance, it requires considerable compute.
res_6129029aInstruction finetuning improves the performance of models with a relatively small amount of compute—for instance, for PaLM 540B, instruction finetuning requires only 0.2% of the pre-training compute but improves the normalized average across evaluation benchmarks by 9.4%.
clm_ce986f92Moreover, smaller models that use instruction finetuning can sometimes outperform larger models without it.
res_3508831bFlan-PaLM achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU.
clm_63f545adAdditionally, we show that instruction finetuning is compatible with a range of model sizes, architectures, and pre-training objectives.
clm_0de0332aTo this end, we publicly release Flan-T5 models that outperform baseline T5 models by a large margin.
res_ba96143924 A.2 Does using CoT prompting for evaluation always improve performance? . . . . . . . . . . . .
res_9c459e4d24 A.3 Does instruction finetuning improve performance more or less for larger models? . . . . . . .
res_f6b9cfffIn Table 5, CoT does not improve performance on MMLU, compared with direct prompting.
res_e775f01aInstruction finetuning improves the normalized average score across four datasets for 8B model by 15.5% and the 540B model by 9.4%.
res_eb2a6704However, the baseline is also higher for the 540B model, so it could also be relevant to compute the relative reduction in error rate, which was 18.4% for the 540B model and 16.6% for the 8B model.
res_565f54c1Table 7 shows that toxicity classification capabilities increase significantly with instruction finetuning, as we observe a consistent improvement both in zero and few-shot classification.
res_6bed31c1All the instruction finetuned models in a zero-shot classification scenario outperform the base models, even when comparing with their performance in the 10-shot scenario.
res_96dd260dHowever, these models are not yet on a par with systems specifically designed for this task, such as Perspective API, which achieves an AUC on 97.0% on the same evaluation dataset.
res_24e0a40bInterestingly, scale does not have a clear linear influence for Flan-PaLM models and Figure 15 shows Flan-T5-XXL outperforms all Flan-PaLM variations.
clm_d3c51e33Interestingly, Flan-T5-XXL outperforms Flan- PaLM models and the influence of scale is not clear and linear.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2210.11416) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.