clm_5924415eIn this work, we present our techniques for training very large transformer models and implement a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of … (excerpt)
arXiv 1909.08053 · cs.CL
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_5924415eIn this work, we present our techniques for training very large transformer models and implement a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of … (excerpt)
res_248cc0f2We sustain 15.1 PetaFLOPs across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline that sustains 39 TeraFLOPs, which is 30% of peak FLOPs.
clm_38338541To demonstrate that large language models can further advance the state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9 … (excerpt)
clm_cc16af95We show that careful attention to the placement of layer normalization in BERT-like models is critical to achieving increased performance as the model size grows.
res_34d008deUsing the GPT-2 model we achieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA accuracy of 63.2%) datasets.
res_4c40c625Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy of 89.4%).
res_a5b43dc0Scaling the model to 8.3 billion parameters on 512 GPUs with 8-way model parallelism, we achieve up to 15.1 PetaFLOPs per second sustained over the entire application.
clm_639a7502We show that the existing BERT architecture results in model degradation as the size increases.
clm_6d93ecb9In addition, we show that our models achieve test set state of the art (SOTA) results on WikiText103, cloze-style prediction accuracy on LAMBADA, and reading comprehension RACE datasets.
res_bcfaf302In summary, our contributions are as follows: • We implement a simple and efficient model parallel approach by making only a few targeted modifications to an existing PyTorch transformer implementation. … (excerpt)
res_16d6443bAdditionally, we use global gradient norm clipping of 1.0 to improve the stability of training large models.
res_ac6350efThis is a strong baseline as it achieves 39 TeraFLOPS during the overall training process, which is 30% of the theoretical peak FLOPS for a single GPU in a DGX-2H … (excerpt)
res_ae0d5670For example, the 8.3 billion parameters case with 8-way (8 GPU) model parallelism achieves 77% of linear scaling.
res_3d2ee114However, even for the largest configuration (8.3 billion parameters) running on 512 GPUs, we achieve 74% scaling relative to linear scaling of the strong single GPU baseline configuration (1.2 billion … (excerpt)
res_dd40023dCompared to the configurations used for our scaling studies in Table 1, the 2.5B model is the same, the 8.3B model has 24 attention heads instead of 32, and the … (excerpt)
res_5616aeaaAs the model size increases, the validation perpelixity decreases and reaches a validation perplexity of 9.27 for the 8.3B model.
res_e196e188Our 8.3B model achieves state of the art perplexity on the WikiText103 test set at a properly adjusted perplexity of 10.81.
res_eb024721At 66.51% accuracy, the 8.3B model similarly surpasses prior cloze accuracy results on the LAMBADA task.
res_c172e935The architecture (b) in Figure 7 eliminates instabilities observed using the original BERT architecture in (a) and also has a lower training loss.
res_56a8565fOn a 3% held-out set, 336M, 1.3B, and 3.9B models achieve validation set perplexity of 1.58, 1.30, and 1.16, respectively, a monotonic decrease with the model size.
res_2c0e56bcFrom Table 5 we observe that (a) as the model size increases, the downstream task performance improves in all cases, (b) our 3.9B model establishes state of the art results … (excerpt)
res_a2f5fc6cWe efficiently trained transformer based models up to 8.3 billion parameter on 512 NVIDIA V100 GPUs with 8-way model parallelism and achieved up to 15.1 PetaFLOPs sustained over the entire … (excerpt)
clm_5e0c92efWe also showed that for BERT models, careful attention to the placement of layer normalization in BERT-like models is critical to achieving increased accuracies as the model size increases.
clm_709bf303We study the effect of model size on down-stream task accuracy and achieve far superior results on downstream tasks and establish new SOTA for WikiText103, LAMBADA, and RACE datasets.
res_3b513fd5Using two GPUs makes training 64% faster.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1909.08053) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.