clm_ec17759bWe propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function.
arXiv 1606.08415 · cs.LG
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_ec17759bWe propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function.
clm_6869bfc8In this work, we introduce a new nonlinearity, the Gaussian Error Linear Unit (GELU).
clm_6fc07e7dWe find that this novel nonlinearity matches or exceeds models with ReLUs or ELUs across tasks from computer vision, natural language processing, and automatic speech recognition.
clm_2e121257In the next section, we show that the GELU exceeds ReLUs and ELUs across numerous tasks.
clm_fea32b26Using these classifiers, we demonstrate in Figure 3 that classifiers using a GELU can be more robust to noised inputs.
res_63230ef3The results in Figure 4 indicate the GELU accommodates different learning rates and significantly outperforms the other nonlinearities.
clm_fc095ee7Next, we demonstrate that for more intricate architectures the GELU nonlinearity again outperforms other nonlinearities.
res_f868ab1eMeanwhile, the GELU achieves a median error of 20.74%, the ReLU obtains 21.77% (without our changes described above, the original 40-4 WideResNet with a ReLU obtains 22.89% (Zagoruyko & Komodakis, … (excerpt)
clm_7ae9a946Across several experiments, the GELU outperformed previous nonlinearities, but it bears semblance to the ReLU and ELU in other respects.
clm_3299f0f7Rather than give credit to this work for the SiLU, the update only cited this work for the GELU so that the “swish” appeared more novel.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1606.08415) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.