MearcAll papers

arXiv 2204.02311 · cs.CL

PaLM: Scaling Language Modeling with Pathways

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, and 61 more

Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.

No reproduction recordedclm_f8940b27
We demonstrate continued benefits of scaling by achieving state-ofthe-art few-shot learning results on hundreds of language understanding and generation benchmarks.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_93c4cb63
On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned stateof-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_3dcba4fb
PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_fd62ca2f
Although these models have achieved near universal state of the art across thousands of natural language tasks, the downside is that they require a significant number of task-specific training examples to finetune the model.

Introduction · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_51f69c0b
In this work, we continue the scaling line of language modeling improvements and train a 540 billion parameter, densely activated, autoregressive Transformer on 780 billion tokens of high-quality text.

Introduction · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_c0fe66ea
This new model, called Pathways Language Model (PaLM), achieves state-of-the-art few-shot results across hundreds of natural language, code, and mathematical reasoning tasks.

Introduction · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1b6bbc2a
With Pathways, we trained a 540B parameter language model on 6144 TPU v4 chips at efficiency levels that could not be reached before for models of this scale.

Introduction · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_21c576e9
In Section 4, we describe how we were able to scale pipeline-free training of PaLM 540B to 6144 chips across two TPU v4 Pods while achieving very high efficiency of 46.2% in model FLOPs utilization (observed throughput relative to theoretical max throughput) and 57.8% in hardware FLOPs utilization. • Continued improvements from scaling – In Section 6, we evaluate PaLM across hundreds of natural language, code, and mathematical reasoning tasks, and achieve state-of-the-art results on the 1We note that these models are all trained either on English data or diverse multilingual sets.

Introduction · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_bd6de7d7
In this work, we demonstrate that when model scaling is combined with chain-of-thought prompting (Wei et al., 2022b), simple few-shot evaluation can outperform or match the finetuned state of the art on a wide array of reasoning tasks.

Introduction · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e40b75f4
In Figure 1 and Section 9, we demonstrate exploratory capabilities in PaLM’s ability to explicitly interpret and explain complex reasoning chains. • Discontinuous improvements – To better understand the scaling behavior, we present results at three different parameter scales: 8B, 62B, and 540B.

Introduction · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d0b540e7
However, for certain tasks, we observe discontinuous improvements, where scaling from 62B to 540B results in a drastic jump in accuracy compared to scaling from 8B to 62B.

Introduction · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4c06f0a8
Even with a relatively small proportion of non-English data (≈ 22%) in the training corpus, few-shot evaluation results from the 540B model are able to bridge the gap with prior finetuned state of the art in non-English summarization tasks and outperform prior state of the art in translation tasks.

Introduction · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9d5dc3db
Firstly, for gender and occupation bias, we found that accuracy on the Winogender coreference task improves with model scale, and PaLM 540B sets a new state-of-the-art result in 1-shot and few-shot settings.

Introduction · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a9f23ad1
Specifically, the standard formulation can be written as: y =x + MLP(LayerNorm(x + Attention(LayerNorm(x))) Whereas the parallel formulation can be written as: y =x + MLP(LayerNorm(x)) + Attention(LayerNorm(x)) The parallel formulation results in roughly 15% faster training speed at large scales, since the MLP and Attention input matrix multiplications can be fused.

Model Architecture · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d2cd91f4
PaLM 540B utilizes the client-server architecture of Pathways to achieve two-way data parallelism at the pod level.

Training Infrastructure · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9efd8720
An interesting aspect of two-way pod-level data parallelism is the challenge of achieving high training throughput for cross-pod gradient transfers at the scale of 6144 TPU v4 chips attached to a total of 1536 hosts across two pods.

Training Infrastructure · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b086aa9d
With these optimizations, we achieve a training throughput of about 1.95x relative to the throughput on a single pod during training (equivalent to 97% of the perfect weak scaling because we double the batch size across two pods relative to a single pod).

Training Infrastructure · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ce7814b3
We propose a new metric for efficiency that is implementation-independent and permits a cleaner comparison of system efficiency, called model FLOPs utilization (MFU).

Training Infrastructure · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c769523c
4 In comparison, PaLM 540B achieves an average training throughput of 238.3K tokens/sec at the batch size of 2048.

Training Infrastructure · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fa220574
PaLM achieves high accelerator utilization because of its parallelism strategy and several other factors, including XLA TPU compiler optimizations, and the use of “parallel layers” (see Section 2).

Training Infrastructure · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_24a6ffc0
PaLM 540B outperforms prior SOTA on 24 of the 29 task in the 1-shot setting and 28 of the 29 tasks in the few-shot setting.

English NLP tasks · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f02e00af
Interestingly, PaLM 540B outperforms prior SOTA by more than 10 points in the few-shot setting on some of the Reading Comprehension and NLI tasks.

English NLP tasks · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ea28a146
While model size has played an important role in achieving these results, PaLM 540B outperforms a similar sized model (Megatron-Turing NLG 530B) on all benchmarks.

English NLP tasks · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_80807f7a
PaLM 540B improves on the average score in both categories by more than 5 points.

English NLP tasks · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5dd8518d
Interestingly, the PaLM 62B outperforms the GPT-3 175B in both categories.

English NLP tasks · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0b4bcd25
As shown in Table 6, PaLM 540B improves the average score of MMLU benchmark by≈ 2 points.

Massive Multitask Language Understanding · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a72c327f
PaLM 540B outperforms the Chinchilla model on all the categories except the category for Other tasks.

Massive Multitask Language Understanding · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_1c37d0e9
On SuperGLUE, we compare with state-of-the-art models such as T5-11B (Raffel et al., 2020) and ST-MoE-32B (Zoph et al., 2022) and show that PaLM obtains competitive close-to-SOTA performance.

Finetuning · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_04061908
It has been shown that such an architecture will generally outperform autoregressive decoder-only models on classification task finetuning, when training cost is equalized (Raffel et al., 2020).

Finetuning · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_db3ce7c1
We show that PaLM is competitive with state-of-the-art while outperforming the best decoder-only autoregressive language model on the leaderboard by a wide margin.

Finetuning · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_24d5b8c8
We see that PaLM significantly outperforms both GPT-3, 7 Gopher, and Chinchilla, and 5-shot PaLM 540B achieves a higher score than the average score of the humans asked to solve the same tasks.

BIG-bench · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6be388a1
PaLM 540B 5-shot outperforms the prior SOTA on 44 out of the 58 common tasks, with per-task results shown in Figure 4.

BIG-bench · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_981e7c55
The results are normalized by setting the maximum score to 100 and the random chance score to 0 for multiple-choice tasks, so that they are negative valued if the model performs worse than random chance.

BIG-bench · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bf52ad17
Positive numbers (blue) indicate that PaLM 540B achieves higher than the prior SOTA, while negative numbers (orange) indicate that the prior SOTA is higher than that of PaLM 540B.

BIG-bench · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7d2843d4
Answer: Yes We can see in Figure 5 that performance on goal step wikihow and logical args follows a log-linear scaling curve, with the PaLM 540B model achieving accuracy close to the best human performance.

BIG-bench · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6ca84123
Performance on english proverbs and logical sequence is also extremely strong, but it follows a discontinuous improvement curve—the improvement from 62B→ 540B is much larger than 8b → 62B.

BIG-bench · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a588ef61
For example, english proverbs requires a very high level of abstract reasoning capability to understand complex metaphors, so the improvement from 25% for PaLM 62B to 87% for PaLM 540B is an extremely exciting result.

BIG-bench · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8346e859
(a) shows log-linear improvements with scale, (b) shows “discontinuous” improvements with scale, and (c) shows relatively flat improvements with scale. tasks had a discontinuity greater than +20%.

BIG-bench · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_be0b1737
For navigate and mathematical induction, PaLM 540B only modestly outperforms PaLM 62B, and both are still far from the best human performance score.

BIG-bench · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_741e44ae
In Figure 6 we show the distribution of improvement over tasks, when comparing PaLM 540B to the average performance score of human evaluations.

BIG-bench · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2204.02311) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.