MearcAll papers

arXiv 2101.03961 · cs.LG cs.AI

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

William Fedus, Barret Zoph, Noam Shazeer

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_493555a7
Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1abfda2d
These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_670a2054
Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the “Colossal Clean Crawled Corpus”, and achieve a 4x speedup over the T5-XXL … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2ed8cb62
8 2.4 Improved Training and Fine-Tuning Techniques . . . . . . . . . . . . . . . .

Switch Transformer 4 · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8d39fcd9
Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain.

Introduction · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_02874bf1
We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher. • Improved pre-training and fine-tuning techniques: (1) selective precision … (excerpt)

Introduction · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1c2823a5
These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.

Introduction · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_3ef90471
We show this simplification preserves model quality, reduces routing computation and performs better.

Simplifying Sparse Routing · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_65fb9912
If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well.

Putting It All Together: The Switch Transformer · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f81abe94
Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k … (excerpt)

Improved Training and Fine-Tuning Techniques · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_68fb2e5c
However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors.

Improved Training and Fine-Tuning Techniques · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d8b1e876
Table 3 measures the improvement of the model quality and reduction of the variance early in training.

Improved Training and Fine-Tuning Techniques · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_82107ca4
Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the … (excerpt)

Improved Training and Fine-Tuning Techniques · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fdf2fc8c
A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better).

Improved Training and Fine-Tuning Techniques · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f5a9e1ca
However, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.

Improved Training and Fine-Tuning Techniques · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d1dcdc8f
Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time.

Scaling Results on a Step-Basis · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cf83fab4
Figure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves.

Scaling Results on a Time-Basis · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_dbe91234
For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline.

Scaling Results on a Time-Basis · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9aaf892f
Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.

Scaling Results on a Time-Basis · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9a351f81
Right Plot: As before, on a wall-clock basis, we find that Switch-Base is still faster, and yields a 2.5x speedup over T5-Large.

Scaling Versus a Larger Dense Model · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_43a21940
Section 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks.

Downstream Results · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4670676e
Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across … (excerpt)

Downstream Results · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_87478f8b
7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy … (excerpt)

Fine-Tuning · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0d2f53c5
Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements … (excerpt)

Fine-Tuning · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_334c291b
Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label.

Distillation · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e5f5396d
We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model.

Distillation · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_885fa62c
At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher’s model quality improvement.

Distillation · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_af044bc7
In Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base.

Multilingual Learning · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4d5c7aaf
We find that we again achieve 30% of the teacher’s performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, … (excerpt)

Multilingual Learning · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5355dc04
9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup.

Multilingual Learning · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d09e78d7
Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages.

Designing Models with Data, Model, and Expert-Parallelism · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_89f599b0
We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.

Designing Models with Data, Model, and Expert-Parallelism · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8983dfbb
Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final … (excerpt)

Designing Models with Data, Model, and Expert-Parallelism · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bd275488
We observe that the Switch- C Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training … (excerpt)

Towards Trillion Parameter Models · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c948fdae
In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively.

Towards Trillion Parameter Models · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_10c4a84d
After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version’s negative log perplexity by over 0.061.

Towards Trillion Parameter Models · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_fd645c42
We note that while the Switch- XXL has state-of-the-art Neg.

Towards Trillion Parameter Models · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_a67041b8
Summing up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality.

Towards Trillion Parameter Models · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_a874b9f1
Do sparse models outperform dense models on the speed-accuracy Pareto curve?

Discussion · page 25 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_be80febc
Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time.

Discussion · page 25 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2101.03961) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.