clm_7f4bcbe8Since this format has a narrower range than single-precision we propose three techniques for preventing the loss of critical information.
arXiv 1710.03740 · cs.AI cs.LG stat.ML
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_7f4bcbe8Since this format has a narrower range than single-precision we propose three techniques for preventing the loss of critical information.
clm_6d27f7eeSecondly, we propose loss-scaling to preserve gradient values with small magnitudes.
clm_5f64c337We demonstrate that the proposed methodology works across a wide variety of tasks and modern large scale (exceeding 100 million parameters) model architectures, trained on large datasets.
clm_464c4e69Using these techniques we demonstrate that a wide variety of network architectures and applications can be trained to match the accuracy FP32 training.
res_bc971ee6Mishra et al. improve on the top-1 accuracy achieved by prior weight and activation quantizations by doubling or tripling the width of layers in popular CNNs.
clm_8f37edc9We demonstrate that this technique works across a variety of applications using state-of-the-art models trained on large scale datasets.
res_cdb85e12This particular network diverges when gradients are not scaled, but scaling them by a factor of 8 (increasing the exponents by 3) is sufficient to match the accuracy achieved with … (excerpt)
res_e8c20686Faster- RCNN was trained on VOC 2007 training set, whereas SSD was trained on a union of VOC 2007 and 2012 data, which is the reason behind baseline mAP difference … (excerpt)
res_ed7af762Model Baseline MP without loss-scale MP with loss-scale Faster R-CNN 69.1% 68.6% 69.7% Multibox SSD 76.9% diverges 77.1% As can be seen in table 2, SSD detector failed to train … (excerpt)
res_1d49161eAs shown in table 3, Pseudo FP16 results are roughly 5 to 10% better than the baseline.
clm_6af455f6Note that we show a randomly selected set of output images, whereas GAN publications typically show a curated set of outputs by excluding poor examples.
clm_cc3c18abFor certain models with a large number of small gradient values, we introduce the gradient scaling method to help them converge to the same accuracy as FP32 baseline models.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1710.03740) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.