MearcAll papers

arXiv 1606.08415 · cs.LG

Gaussian Error Linear Units (GELUs)

Dan Hendrycks, Kevin Gimpel

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_ec17759b
We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_6869bfc8
In this work, we introduce a new nonlinearity, the Gaussian Error Linear Unit (GELU).

I NTRODUCTION · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_6fc07e7d
We find that this novel nonlinearity matches or exceeds models with ReLUs or ELUs across tasks from computer vision, natural language processing, and automatic speech recognition.

I NTRODUCTION · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_2e121257
In the next section, we show that the GELU exceeds ReLUs and ELUs across numerous tasks.

GELU · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_fea32b26
Using these classifiers, we demonstrate in Figure 3 that classifiers using a GELU can be more robust to noised inputs.

MNIST C LASSIFICATION · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_63230ef3
The results in Figure 4 indicate the GELU accommodates different learning rates and significantly outperforms the other nonlinearities.

MNIST A UTOENCODER · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_fc095ee7
Next, we demonstrate that for more intricate architectures the GELU nonlinearity again outperforms other nonlinearities.

CIFAR-10/100 C LASSIFICATION · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f868ab1e
Meanwhile, the GELU achieves a median error of 20.74%, the ReLU obtains 21.77% (without our changes described above, the original 40-4 WideResNet with a ReLU obtains 22.89% (Zagoruyko & Komodakis, … (excerpt)

CIFAR-10/100 C LASSIFICATION · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7ae9a946
Across several experiments, the GELU outperformed previous nonlinearities, but it bears semblance to the ReLU and ELU in other respects.

D ISCUSSION · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_3299f0f7
Rather than give credit to this work for the SiLU, the update only cited this work for the GELU so that the “swish” appeared more novel.

H ISTORY OF THE GELU AND SILU · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1606.08415) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.