clm_493555a7Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats.
arXiv 2101.03961 · cs.LG cs.AI
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_493555a7Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats.
res_1abfda2dThese improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages.
res_670a2054Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the “Colossal Clean Crawled Corpus”, and achieve a 4x speedup over the T5-XXL … (excerpt)
res_2ed8cb628 2.4 Improved Training and Fine-Tuning Techniques . . . . . . . . . . . . . . . .
res_8d39fcd9Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain.
res_02874bf1We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher. • Improved pre-training and fine-tuning techniques: (1) selective precision … (excerpt)
res_1c2823a5These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.
clm_3ef90471We show this simplification preserves model quality, reduces routing computation and performs better.
clm_65fb9912If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well.
res_f81abe94Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k … (excerpt)
clm_68fb2e5cHowever, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors.
res_d8b1e876Table 3 measures the improvement of the model quality and reduction of the variance early in training.
res_82107ca4Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the … (excerpt)
res_fdf2fc8cA sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better).
res_f5a9e1caHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.
res_d1dcdc8fOur Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time.
res_cf83fab4Figure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves.
clm_dbe91234For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline.
res_9aaf892fOur 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.
res_9a351f81Right Plot: As before, on a wall-clock basis, we find that Switch-Base is still faster, and yields a 2.5x speedup over T5-Large.
res_43a21940Section 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks.
res_4670676eFinally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across … (excerpt)
res_87478f8b7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy … (excerpt)
res_0d2f53c5Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements … (excerpt)
res_334c291bFurthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label.
res_e5f5396dWe can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model.
res_885fa62cAt the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher’s model quality improvement.
res_af044bc7In Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base.
res_4d5c7aafWe find that we again achieve 30% of the teacher’s performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, … (excerpt)
res_5355dc049 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup.
res_d09e78d7Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages.
res_89f599b0We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.
res_8983dfbbOver all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final … (excerpt)
res_bd275488We observe that the Switch- C Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training … (excerpt)
res_c948fdaeIn the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively.
res_10c4a84dAfter 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version’s negative log perplexity by over 0.061.
clm_fd645c42We note that while the Switch- XXL has state-of-the-art Neg.
clm_a67041b8Summing up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality.
clm_a874b9f1Do sparse models outperform dense models on the speed-accuracy Pareto curve?
clm_be80febcAcross a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2101.03961) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.