MearcAll papers

arXiv 1701.06538 · cs.LG cs.CL cs.NE

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and 1 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedres_a861f6e6
In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency … (excerpt)

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_3d9c08fa
On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9713cb90
We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.

C ONDITIONAL COMPUTATION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ee53bac7
The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the … (excerpt)

E XPERIMENTS · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b9071ec3
Even the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.

E XPERIMENTS · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_dc25362c
The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the … (excerpt)

E XPERIMENTS · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0e3f5e26
When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at … (excerpt)

E XPERIMENTS · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b0c27c16
Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT’14 En →Fr and En→De benchmarks.

M ACHINE TRANSLATION (SINGLE LANGUAGE PAIR ) · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e7f17ecb
The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.

M ACHINE TRANSLATION (SINGLE LANGUAGE PAIR ) · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_660bb926
The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model.

M ULTILINGUAL MACHINE TRANSLATION · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3133e807
On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual … (excerpt)

M ULTILINGUAL MACHINE TRANSLATION · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_0fca435a
We look forward to seeing many novel implementations and applications of conditional computation in the years to come.

C ONCLUSION · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_561a1dfd
The two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful.

M ORE EXPENSIVE MODELS · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5befa845
Comparing after 10 epochs, our model has a lower test perplexity by 18%.

M ORE EXPENSIVE MODELS · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d6d78bdc
Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE 16 Under review as a conference paper at ICLR 2017 model than for the baseline model.

M ORE EXPENSIVE MODELS · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cd552732
As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.

M ACHINE TRANSLATION - E XPERIMENTAL DETAILS · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_c73bd63f
Mbatchwise(X,m )j,i = {1 ifXj,i is in the topm values for to experti 0 otherwise (18) As our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a … (excerpt)

S TRICTLY BALANCED GATING · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1701.06538) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.