MearcAll papers

arXiv 1910.02054 · cs.LG cs.DC stat.ML

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_80b4c350
We develop a novel solution, Zero Redundancy Optimizer (ZeRO), to optimize memory, vastly improving training speed while increasing the model size that can be efficiently trained.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ecbe505f
We implement and evaluate ZeRO: it trains large models of over 100B parameter with super-linear speedup on 400 GPUs, achieving throughput of 15 Petaflops.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2ca7cfe8
This represents an 8x increase in model size and 10x increase in achievable performance over state-of-the-art.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_11df0fd5
We tested a 40B parameter model using Megatron-LM across two DGX-2 nodes and observe about 5 T f lops per V100 GPU (less than 5% of hardware peak).

Extended Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_c8a2c204
We show that ZeRO can be combined with MP, resulting in a max theoretical memory reduction of Nd × Nm times on each device with a DP degree of Nd … (excerpt)

Extended Introduction · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_09a1b79b
Therefore, our focus for this implementation is to efficiently support models with 10x parameters ( ∼100B parameters) than state-of-the-art (SOTA) while still being within reach of the compute capabilities of … (excerpt)

Extended Introduction · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_53b84dc4
We implement and evaluate a subset of optimizations in ZeRO called ZeRO-100B — Pos+g of ZeRO-DP plus ZeRO-R — that allow us to achieve this goal.

Extended Introduction · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fcf7d962
This is more than 10x improvement in training speed compared to SOTA for the same model size.

Extended Introduction · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bf8b2d03
Scalability We observe super linear speedup in the regime of 64-400 GPUs, where the performance more than doubles when we double the number of GPUs.

Extended Introduction · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_33684d3f
We expect this behaviour to continue further as we increase the number of GPUs beyond 400.

Extended Introduction · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8abc2ab2
On rare cases, ZeRO-R may offload just the activation checkpoints for very large models to improve performance (see Sec. 6.1 for details).

Reducing Activation Memory · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8096ce85
Our results show that ZeRO-100B can efficiently train models with up to 170B parameters, 8x bigger than SOTA, up to 10x faster and with improved usability.

Implementation and Evaluation · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8d319b82
ZeRO-100B achieves a sustained throughput of 15 PetaFlops (over 30% of the peak) on average for models with 8B to 100B parameters.

Speed and Model Size · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cff862ca
In comparison, the baseline MP performance degrades quickly with the increase in model size: MP incurs high communication volume between GPUs, and going beyond a single node to fit larger … (excerpt)

Speed and Model Size · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5fbae6d2
ZeRO-100B achieves up to 10x speedup over baseline, significantly outperforming on large models.

Speed and Model Size · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_49485800
We expect the performance to improve as we increase the number of GPUs due to super-linear speedup of ZeRO-100B as we discuss next.

Speed and Model Size · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_828733a2
Fig. 4 shows that ZeRO-100B can train models with up to 13B parameters without MP on 128 GPUs, achieving throughput over 40 TFlops per GPU on average.

Democratizing Large Model Training · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_02d4e23e
Max Achievable Performance Figure 8 shows the best achievable performance for different set of optimizations.

Memory and Performance Analysis · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d801e4d2
It achieved the new SOTA for language models with Webtext-103 perplexity of 10.21.

Turing-NLG, the SOTA language model with 17B parameters · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6b9164c3
ZeRO- 100B achieves a sustained throughput of 41.4 TFlops/GPU for this model.

Turing-NLG, the SOTA language model with 17B parameters · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d80168f7
While our implementation, ZeRO-100B, enables 8x increase in model sizes, over 10x in throughput improvement, achieves super-linear speedups on modern GPU clusters, and trains the largest model in the world, … (excerpt)

Concluding Remarks · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1910.02054) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.