clm_f8940b27We demonstrate continued benefits of scaling by achieving state-ofthe-art few-shot learning results on hundreds of language understanding and generation benchmarks.
arXiv 2204.02311 · cs.CL
Read on arXiv · CC BY 4.0
Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.
clm_f8940b27We demonstrate continued benefits of scaling by achieving state-ofthe-art few-shot learning results on hundreds of language understanding and generation benchmarks.
res_93c4cb63On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned stateof-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark.
clm_3dcba4fbPaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks.
clm_fd62ca2fAlthough these models have achieved near universal state of the art across thousands of natural language tasks, the downside is that they require a significant number of task-specific training examples to finetune the model.
res_51f69c0bIn this work, we continue the scaling line of language modeling improvements and train a 540 billion parameter, densely activated, autoregressive Transformer on 780 billion tokens of high-quality text.
clm_c0fe66eaThis new model, called Pathways Language Model (PaLM), achieves state-of-the-art few-shot results across hundreds of natural language, code, and mathematical reasoning tasks.
res_1b6bbc2aWith Pathways, we trained a 540B parameter language model on 6144 TPU v4 chips at efficiency levels that could not be reached before for models of this scale.
res_21c576e9In Section 4, we describe how we were able to scale pipeline-free training of PaLM 540B to 6144 chips across two TPU v4 Pods while achieving very high efficiency of 46.2% in model FLOPs utilization (observed throughput relative to theoretical max throughput) and 57.8% in hardware FLOPs utilization. • Continued improvements from scaling – In Section 6, we evaluate PaLM across hundreds of natural language, code, and mathematical reasoning tasks, and achieve state-of-the-art results on the 1We note that these models are all trained either on English data or diverse multilingual sets.
clm_bd6de7d7In this work, we demonstrate that when model scaling is combined with chain-of-thought prompting (Wei et al., 2022b), simple few-shot evaluation can outperform or match the finetuned state of the art on a wide array of reasoning tasks.
res_e40b75f4In Figure 1 and Section 9, we demonstrate exploratory capabilities in PaLM’s ability to explicitly interpret and explain complex reasoning chains. • Discontinuous improvements – To better understand the scaling behavior, we present results at three different parameter scales: 8B, 62B, and 540B.
res_d0b540e7However, for certain tasks, we observe discontinuous improvements, where scaling from 62B to 540B results in a drastic jump in accuracy compared to scaling from 8B to 62B.
res_4c06f0a8Even with a relatively small proportion of non-English data (≈ 22%) in the training corpus, few-shot evaluation results from the 540B model are able to bridge the gap with prior finetuned state of the art in non-English summarization tasks and outperform prior state of the art in translation tasks.
res_9d5dc3dbFirstly, for gender and occupation bias, we found that accuracy on the Winogender coreference task improves with model scale, and PaLM 540B sets a new state-of-the-art result in 1-shot and few-shot settings.
res_a9f23ad1Specifically, the standard formulation can be written as: y =x + MLP(LayerNorm(x + Attention(LayerNorm(x))) Whereas the parallel formulation can be written as: y =x + MLP(LayerNorm(x)) + Attention(LayerNorm(x)) The parallel formulation results in roughly 15% faster training speed at large scales, since the MLP and Attention input matrix multiplications can be fused.
res_d2cd91f4PaLM 540B utilizes the client-server architecture of Pathways to achieve two-way data parallelism at the pod level.
res_9efd8720An interesting aspect of two-way pod-level data parallelism is the challenge of achieving high training throughput for cross-pod gradient transfers at the scale of 6144 TPU v4 chips attached to a total of 1536 hosts across two pods.
res_b086aa9dWith these optimizations, we achieve a training throughput of about 1.95x relative to the throughput on a single pod during training (equivalent to 97% of the perfect weak scaling because we double the batch size across two pods relative to a single pod).
clm_ce7814b3We propose a new metric for efficiency that is implementation-independent and permits a cleaner comparison of system efficiency, called model FLOPs utilization (MFU).
res_c769523c4 In comparison, PaLM 540B achieves an average training throughput of 238.3K tokens/sec at the batch size of 2048.
res_fa220574PaLM achieves high accelerator utilization because of its parallelism strategy and several other factors, including XLA TPU compiler optimizations, and the use of “parallel layers” (see Section 2).
res_24a6ffc0PaLM 540B outperforms prior SOTA on 24 of the 29 task in the 1-shot setting and 28 of the 29 tasks in the few-shot setting.
res_f02e00afInterestingly, PaLM 540B outperforms prior SOTA by more than 10 points in the few-shot setting on some of the Reading Comprehension and NLI tasks.
res_ea28a146While model size has played an important role in achieving these results, PaLM 540B outperforms a similar sized model (Megatron-Turing NLG 530B) on all benchmarks.
res_80807f7aPaLM 540B improves on the average score in both categories by more than 5 points.
res_5dd8518dInterestingly, the PaLM 62B outperforms the GPT-3 175B in both categories.
res_0b4bcd25As shown in Table 6, PaLM 540B improves the average score of MMLU benchmark by≈ 2 points.
res_a72c327fPaLM 540B outperforms the Chinchilla model on all the categories except the category for Other tasks.
clm_1c37d0e9On SuperGLUE, we compare with state-of-the-art models such as T5-11B (Raffel et al., 2020) and ST-MoE-32B (Zoph et al., 2022) and show that PaLM obtains competitive close-to-SOTA performance.
clm_04061908It has been shown that such an architecture will generally outperform autoregressive decoder-only models on classification task finetuning, when training cost is equalized (Raffel et al., 2020).
clm_db3ce7c1We show that PaLM is competitive with state-of-the-art while outperforming the best decoder-only autoregressive language model on the leaderboard by a wide margin.
res_24d5b8c8We see that PaLM significantly outperforms both GPT-3, 7 Gopher, and Chinchilla, and 5-shot PaLM 540B achieves a higher score than the average score of the humans asked to solve the same tasks.
res_6be388a1PaLM 540B 5-shot outperforms the prior SOTA on 44 out of the 58 common tasks, with per-task results shown in Figure 4.
res_981e7c55The results are normalized by setting the maximum score to 100 and the random chance score to 0 for multiple-choice tasks, so that they are negative valued if the model performs worse than random chance.
res_bf52ad17Positive numbers (blue) indicate that PaLM 540B achieves higher than the prior SOTA, while negative numbers (orange) indicate that the prior SOTA is higher than that of PaLM 540B.
res_7d2843d4Answer: Yes We can see in Figure 5 that performance on goal step wikihow and logical args follows a log-linear scaling curve, with the PaLM 540B model achieving accuracy close to the best human performance.
res_6ca84123Performance on english proverbs and logical sequence is also extremely strong, but it follows a discontinuous improvement curve—the improvement from 62B→ 540B is much larger than 8b → 62B.
res_a588ef61For example, english proverbs requires a very high level of abstract reasoning capability to understand complex metaphors, so the improvement from 25% for PaLM 62B to 87% for PaLM 540B is an extremely exciting result.
res_8346e859(a) shows log-linear improvements with scale, (b) shows “discontinuous” improvements with scale, and (c) shows relatively flat improvements with scale. tasks had a discontinuity greater than +20%.
res_be0b1737For navigate and mathematical induction, PaLM 540B only modestly outperforms PaLM 62B, and both are still far from the best human performance score.
res_741e44aeIn Figure 6 we show the distribution of improvement over tasks, when comparing PaLM 540B to the average performance score of human evaluations.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2204.02311) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.