MearcAll papers

arXiv 1909.08053 · cs.CL

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_5924415e
In this work, we present our techniques for training very large transformer models and implement a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_248cc0f2
We sustain 15.1 PetaFLOPs across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline that sustains 39 TeraFLOPs, which is 30% of peak FLOPs.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_38338541
To demonstrate that large language models can further advance the state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9 … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_cc16af95
We show that careful attention to the placement of layer normalization in BERT-like models is critical to achieving increased performance as the model size grows.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_34d008de
Using the GPT-2 model we achieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA accuracy of 63.2%) datasets.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4c40c625
Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy of 89.4%).

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a5b43dc0
Scaling the model to 8.3 billion parameters on 512 GPUs with 8-way model parallelism, we achieve up to 15.1 PetaFLOPs per second sustained over the entire application.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_639a7502
We show that the existing BERT architecture results in model degradation as the size increases.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_6d93ecb9
In addition, we show that our models achieve test set state of the art (SOTA) results on WikiText103, cloze-style prediction accuracy on LAMBADA, and reading comprehension RACE datasets.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bcfaf302
In summary, our contributions are as follows: • We implement a simple and efficient model parallel approach by making only a few targeted modifications to an existing PyTorch transformer implementation. … (excerpt)

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_16d6443b
Additionally, we use global gradient norm clipping of 1.0 to improve the stability of training large models.

Training Optimization and Hyperparameters · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ac6350ef
This is a strong baseline as it achieves 39 TeraFLOPS during the overall training process, which is 30% of the theoretical peak FLOPS for a single GPU in a DGX-2H … (excerpt)

M ODEL AND DATA PARALLELISM · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ae0d5670
For example, the 8.3 billion parameters case with 8-way (8 GPU) model parallelism achieves 77% of linear scaling.

M ODEL AND DATA PARALLELISM · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3d2ee114
However, even for the largest configuration (8.3 billion parameters) running on 512 GPUs, we achieve 74% scaling relative to linear scaling of the strong single GPU baseline configuration (1.2 billion … (excerpt)

M ODEL AND DATA PARALLELISM · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_dd40023d
Compared to the configurations used for our scaling studies in Table 1, the 2.5B model is the same, the 8.3B model has 24 attention heads instead of 32, and the … (excerpt)

Language Modeling Results Using GPT-2 · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5616aeaa
As the model size increases, the validation perpelixity decreases and reaches a validation perplexity of 9.27 for the 8.3B model.

Language Modeling Results Using GPT-2 · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e196e188
Our 8.3B model achieves state of the art perplexity on the WikiText103 test set at a properly adjusted perplexity of 10.81.

Language Modeling Results Using GPT-2 · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_eb024721
At 66.51% accuracy, the 8.3B model similarly surpasses prior cloze accuracy results on the LAMBADA task.

Language Modeling Results Using GPT-2 · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c172e935
The architecture (b) in Figure 7 eliminates instabilities observed using the original BERT architecture in (a) and also has a lower training loss.

Bi-directional Transformer Results Using BERT · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_56a8565f
On a 3% held-out set, 336M, 1.3B, and 3.9B models achieve validation set perplexity of 1.58, 1.30, and 1.16, respectively, a monotonic decrease with the model size.

Model · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2c0e56bc
From Table 5 we observe that (a) as the model size increases, the downstream task performance improves in all cases, (b) our 3.9B model establishes state of the art results … (excerpt)

Model · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a2f5fc6c
We efficiently trained transformer based models up to 8.3 billion parameter on 512 NVIDIA V100 GPUs with 8-way model parallelism and achieved up to 15.1 PetaFLOPs sustained over the entire … (excerpt)

Conclusion and Future Work · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_5e0c92ef
We also showed that for BERT models, careful attention to the placement of layer normalization in BERT-like models is critical to achieving increased accuracies as the model size increases.

Conclusion and Future Work · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_709bf303
We study the effect of model size on down-stream task accuracy and achieve far superior results on downstream tasks and establish new SOTA for WikiText103, LAMBADA, and RACE datasets.

Conclusion and Future Work · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3b513fd5
Using two GPUs makes training 64% faster.

Strong Scaling · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1909.08053) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.