res_a861f6e6In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency … (excerpt)
arXiv 1701.06538 · cs.LG cs.CL cs.NE
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
res_a861f6e6In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency … (excerpt)
clm_3d9c08faOn large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.
res_9713cb90We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.
res_ee53bac7The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the … (excerpt)
res_b9071ec3Even the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.
clm_dc25362cThe operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the … (excerpt)
res_0e3f5e26When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at … (excerpt)
res_b0c27c16Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT’14 En →Fr and En→De benchmarks.
res_e7f17ecbThe perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.
res_660bb926The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model.
res_3133e807On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual … (excerpt)
clm_0fca435aWe look forward to seeing many novel implementations and applications of conditional computation in the years to come.
res_561a1dfdThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful.
res_5befa845Comparing after 10 epochs, our model has a lower test perplexity by 18%.
res_d6d78bdcPerplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE 16 Under review as a conference paper at ICLR 2017 model than for the baseline model.
res_cd552732As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.
clm_c73bd63fMbatchwise(X,m )j,i = {1 ifXj,i is in the topm values for to experti 0 otherwise (18) As our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a … (excerpt)
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1701.06538) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.