clm_95af5cfdL2 regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is not the case for adaptive … (excerpt)
arXiv 1711.05101 · cs.LG cs.NE math.OC
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_95af5cfdL2 regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is not the case for adaptive … (excerpt)
clm_f3683946While common implementations of these algorithms employ L 2 regularization (often calling it “weight decay” in what may be misleading due to the inequivalence we expose), we propose a simple … (excerpt)
clm_ccbd6deeWe provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam … (excerpt)
clm_e233cd40We show that a major factor of the poor generalization of the most popular adaptive gradient method, Adam, is due to the fact that L2 regularization is not nearly as … (excerpt)
res_2af5de35One possible explanation why Adam and other adaptive gradient methods might be outperformed by SGD with momentum is that common deep learning libraries only implement L 2 regularization, not the … (excerpt)
res_49bd6e03In a comprehensive analysis, we show that Adam generalizes substantially better with decoupled weight decay than with L2 regularization, achieving 15% relative improvement in test error (see Figures 2 and … (excerpt)
clm_b31b59b8We demonstrate this formally for a simple special case of adaptive gradient algorithm with a fixed preconditioner: Proposition 3 (Weight decay = scale-adjusted L2 reg for adaptive gradient algorithm with … (excerpt)
res_d9b883eaOur experimental setup follows that of Gastaldi (2017), who proposed, in addition to L 2 regularization, to apply the new Shake-Shake regularization to a 3-branch residual DNN that allowed to … (excerpt)
clm_7fdfeb28Since Adam already adapts its parameterwise learning rates it is not as common to use a learning rate multiplier schedule with it as it is with SGD, but as our … (excerpt)
res_e2ffacfdFigure 1 shows that decoupled weight decay outperforms L2 regularization for all learning rate schedules, with larger differences for better learning rate schedules.
clm_2d426578The figure also shows that cosine annealing clearly outperforms the other learning rate schedules; we thus used cosine annealing for the remainder of the experiments.
res_9fdf3890Consider, e.g., the setting at the top left black circle (α = 1/2,λ = 1/8∗ 0.001); only changing either α orλ by itself would worsen results, while changing both of … (excerpt)
res_a3fbaf86In summary, the results in Figure 2 support our hypothesis that the weight decay and learning rate hyperparameters can be decoupled, and that this in turn simplifies the problem of … (excerpt)
res_c3747268The results on ImageNet32x32 (see SuppFigure 4 in the Appendix) yield the same conclusion of substantially improved generalization performance.
res_b18cc462For the default learning rate of 0.001, AdamW achieved 15% relative improvement in test error compared to Adam both on CIFAR-10 (also see SuppFigure 5) and ImageNet32x32 (also see SuppFigure … (excerpt)
res_34c5be19These improvements closed most of the gap between Adam and SGDWR on CIFAR-10 and yielded comparable performance on ImageNet32x32.
res_bb02c7a3AdamW with the same hyperparameter setting as Adam yielded higher test set accuracy on Deep4Net (73.68% versus 71.37%) and statistically significantly higher test set accuracy on ResNet (72.04% versus 61.34%).
clm_69a7dea2Here, we propose to reduce this dependence by normalizing the values of weight decay.
clm_eec2b9caSGDR has been successfully adopted to lead to new state-of-the-art results for popular image classification benchmarks (Huang et al., 2017; Gastaldi, 2017; Zoph et al., 2017), and we therefore already … (excerpt)
res_554c2464(15) In order to achieve good anytime performance, one can start with an initially small Ti (e.g., from 1% to 10% of the expected total budget) and multiply it by … (excerpt)
res_a4417327These results are not very surprising given Figure 1 in the main paper (which demonstrates both the improvements possible by using some learning rate schedule, such as cosine annealing, and … (excerpt)
res_b94e3bedIn contrast, had we used the same raw weight decay valuesλ for ImageNet32x32 as for CIFAR- 10 and for the same number of epochs, without the proposed normalization, λ would … (excerpt)
res_6c2c7cc3The results show that Adam and its variants with decoupled weight decay converge faster (in terms of training loss) on CIFAR-10 than the corresponding SGD variants (the difference for ImageNet32x32 … (excerpt)
res_e054a80dInterestingly, SuppFigure 5 and SuppFigure 6 show that the restart variants AdamWR and SGDWR also demonstrate better generalization than AdamW and SGDW, respectively.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1711.05101) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.