MearcAll papers

arXiv 1502.03167 · cs.LG

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Sergey Ioffe, Christian Szegedy

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedres_9891d249
Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f994c706
Using an ensemble of batchnormalized networks, we improve upon the best published result on ImageNet classification: reaching 4.9% top-5 validation error (and 4.8% test error), exceeding the accuracy of human … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_bb73ecb4
We propose a new mechanism, which we call Batch Normalization , that takes a step towards reducing internal covariate shift, and in doing so dramatically accelerates the training of deep … (excerpt)

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1ddeb463
In Sec. 4.2, we apply Batch Normalization to the bestperforming ImageNet classification network, and show that we can match its performance using only 7% of the training steps, and can … (excerpt)

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7fcf9f8d
Using an ensemble of such networks trained with Batch Normalization, we achieve the top-5 error rate that improves upon the best known results on ImageNet classification.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_b0cfe92e
We were interested in the comparison between the baseline and batch-normalized networks, rather than achieving the state of the art performance on MNIST (which the described architecture does not).

Activations over time · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d2e349a5
In a batch-normalized model, we have been able to achieve a training speedup from higher learning rates, with no ill side effects (Sec. 3.3).

Accelerating BN Networks · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7302089e
This led to about 1% improvements in the validation accuracy, which is consistent with the view of Batch Normalization as a regularizer (Sec. 3.4): the randomization inherent in our method … (excerpt)

Accelerating BN Networks · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fedd5995
Inception reached the accuracy of 72.2% after 31· 106 training steps.

Accelerating BN Networks · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ad995ba0
The Figure 3 shows, for each network, the number of training steps required to reach the same 72.2% accuracy, as well as the maximum validation accuracy reached by the network … (excerpt)

Accelerating BN Networks · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fac092b5
BN-x5 needs 14 times fewer steps than Inception to reach the 72.2% accuracy.

Accelerating BN Networks · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4c023818
It reaches 74.8% after6·106 steps, i.e. 5 times fewer steps than required by Inception to reach 72.2%.

Accelerating BN Networks · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5ff966fb
Indeed, BN-x5-Sigmoid achieves the accuracy of 69.8%.

Accelerating BN Networks · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1433dc07
Without Batch Normalization, Inception with sigmoid never achieves better than 1/1000 accuracy.

Accelerating BN Networks · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_28f5b1d3
Each network achieved its maximum accuracy after about 6· 106 training steps.

Accelerating BN Networks · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_db22bd1c
We have presented a novel mechanism for dramatically accelerating the training of deep networks.

Conclusion · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_3b90d9d2
By further increasing the learning rates, removing Dropout, and applying other modifications afforded by Batch Normalization, we reach the previous state of the art with only a small fraction of … (excerpt)

Conclusion · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_fc67fa7f
The goal of Batch Normalization is to achieve a stable distribution of activation values throughout training, and in our experiments we apply it before the nonlinearity since that is where … (excerpt)

Conclusion · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_1eaf0885
Finally, we believe that further theoretical analysis of the algorithm would allow still more improvements and applications.

Conclusion · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1502.03167) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.