MearcAll papers

arXiv 1710.03740 · cs.AI cs.LG stat.ML

Mixed Precision Training

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, and 5 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_7f4bcbe8
Since this format has a narrower range than single-precision we propose three techniques for preventing the loss of critical information.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_6d27f7ee
Secondly, we propose loss-scaling to preserve gradient values with small magnitudes.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_5f64c337
We demonstrate that the proposed methodology works across a wide variety of tasks and modern large scale (exceeding 100 million parameters) model architectures, trained on large datasets.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_464c4e69
Using these techniques we demonstrate that a wide variety of network architectures and applications can be trained to match the accuracy FP32 training.

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bc971ee6
Mishra et al. improve on the top-1 accuracy achieved by prior weight and activation quantizations by doubling or tripling the width of layers in popular CNNs.

R ELATED WORK · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_8f37edc9
We demonstrate that this technique works across a variety of applications using state-of-the-art models trained on large scale datasets.

R ELATED WORK · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cdb85e12
This particular network diverges when gradients are not scaled, but scaling them by a factor of 8 (increasing the exponents by 3) is sufficient to match the accuracy achieved with … (excerpt)

L OSS SCALING · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e8c20686
Faster- RCNN was trained on VOC 2007 training set, whereas SSD was trained on a union of VOC 2007 and 2012 data, which is the reason behind baseline mAP difference … (excerpt)

D ETECTION CNN S · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ed7af762
Model Baseline MP without loss-scale MP with loss-scale Faster R-CNN 69.1% 68.6% 69.7% Multibox SSD 76.9% diverges 77.1% As can be seen in table 2, SSD detector failed to train … (excerpt)

D ETECTION CNN S · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1d49161e
As shown in table 3, Pseudo FP16 results are roughly 5 to 10% better than the baseline.

S PEECH RECOGNITION · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_6af455f6
Note that we show a randomly selected set of output images, whereas GAN publications typically show a curated set of outputs by excluding poor examples.

DCGAN RESULTS · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_cc3c18ab
For certain models with a large number of small gradient values, we introduce the gradient scaling method to help them converge to the same accuracy as FP32 baseline models.

C ONCLUSIONS AND FUTURE WORK · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1710.03740) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.