MearcAll papers

arXiv 1607.06450 · stat.ML cs.LG

Layer Normalization

Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_3d577b57
Empirically, we show that layer normalization can substantially reduce the training time compared with previously published techniques.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_2232e97f
We show that layer normalization works well for RNNs and improves both the training time and the generalization performance of several existing RNN models.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_8f83fd9b
We show that the normalization scalarσ can implicitly reduce learning rate and makes learning more stable.

Geometry of parameter space during learning · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_a8d3901e
We show that Riemannian metric along the magnitude of the incoming weights for the standard GLM is scaled by the norm of its input, whereas learning the gain parameters for … (excerpt)

The geometry of normalized generalized linear models · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6d5e6f7b
After every 300 iterations, we compute Recall@K (R@K) values on a held out validation set and save the model whenever R@K improves.

Order embeddings of images and language · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c755dc68
In Table 2, the test set results are reported from which we observe that layer normalization also results in improved generalization over the original model.

Order embeddings of images and language · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_fe4a2821
The results we report are state-of-the-art for RNN embedding models, with only the structure-preserving model of Wang et al. [2016] reporting better results on this task.

Order embeddings of images and language · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fc0bea37
We observe that applying layer normalization results both in speedup over the baseline as well as better final results after 1M iterations are performed as shown in Table 3.

Skip-thought vectors · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_15132859
We show how layer normalization compares with batch normalization on the well-studied permutation invariant MNIST classification problem.

Permutation invariant MNIST · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_1f693ee4
In our preliminary experiments, we observed that layer normalization offers a speedup over the baseline model without normalization, but batch normalization outperforms the other methods.

Convolutional Networks · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_34ba3bbc
In this paper, we introduced layer normalization to speed-up the training of neural networks.

Conclusion · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1607.06450) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.