clm_80b4c350We develop a novel solution, Zero Redundancy Optimizer (ZeRO), to optimize memory, vastly improving training speed while increasing the model size that can be efficiently trained.
arXiv 1910.02054 · cs.LG cs.DC stat.ML
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_80b4c350We develop a novel solution, Zero Redundancy Optimizer (ZeRO), to optimize memory, vastly improving training speed while increasing the model size that can be efficiently trained.
res_ecbe505fWe implement and evaluate ZeRO: it trains large models of over 100B parameter with super-linear speedup on 400 GPUs, achieving throughput of 15 Petaflops.
res_2ca7cfe8This represents an 8x increase in model size and 10x increase in achievable performance over state-of-the-art.
res_11df0fd5We tested a 40B parameter model using Megatron-LM across two DGX-2 nodes and observe about 5 T f lops per V100 GPU (less than 5% of hardware peak).
clm_c8a2c204We show that ZeRO can be combined with MP, resulting in a max theoretical memory reduction of Nd × Nm times on each device with a DP degree of Nd … (excerpt)
res_09a1b79bTherefore, our focus for this implementation is to efficiently support models with 10x parameters ( ∼100B parameters) than state-of-the-art (SOTA) while still being within reach of the compute capabilities of … (excerpt)
res_53b84dc4We implement and evaluate a subset of optimizations in ZeRO called ZeRO-100B — Pos+g of ZeRO-DP plus ZeRO-R — that allow us to achieve this goal.
res_fcf7d962This is more than 10x improvement in training speed compared to SOTA for the same model size.
res_bf8b2d03Scalability We observe super linear speedup in the regime of 64-400 GPUs, where the performance more than doubles when we double the number of GPUs.
res_33684d3fWe expect this behaviour to continue further as we increase the number of GPUs beyond 400.
res_8abc2ab2On rare cases, ZeRO-R may offload just the activation checkpoints for very large models to improve performance (see Sec. 6.1 for details).
res_8096ce85Our results show that ZeRO-100B can efficiently train models with up to 170B parameters, 8x bigger than SOTA, up to 10x faster and with improved usability.
res_8d319b82ZeRO-100B achieves a sustained throughput of 15 PetaFlops (over 30% of the peak) on average for models with 8B to 100B parameters.
res_cff862caIn comparison, the baseline MP performance degrades quickly with the increase in model size: MP incurs high communication volume between GPUs, and going beyond a single node to fit larger … (excerpt)
res_5fbae6d2ZeRO-100B achieves up to 10x speedup over baseline, significantly outperforming on large models.
res_49485800We expect the performance to improve as we increase the number of GPUs due to super-linear speedup of ZeRO-100B as we discuss next.
res_828733a2Fig. 4 shows that ZeRO-100B can train models with up to 13B parameters without MP on 128 GPUs, achieving throughput over 40 TFlops per GPU on average.
res_02d4e23eMax Achievable Performance Figure 8 shows the best achievable performance for different set of optimizations.
res_d801e4d2It achieved the new SOTA for language models with Webtext-103 perplexity of 10.21.
res_6b9164c3ZeRO- 100B achieves a sustained throughput of 41.4 TFlops/GPU for this model.
res_d80168f7While our implementation, ZeRO-100B, enables 8x increase in model sizes, over 10x in throughput improvement, achieves super-linear speedups on modern GPU clusters, and trains the largest model in the world, … (excerpt)
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1910.02054) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.