MearcAll papers

arXiv 1711.05101 · cs.LG cs.NE math.OC

Decoupled Weight Decay Regularization

Ilya Loshchilov, Frank Hutter

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_95af5cfd
L2 regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is not the case for adaptive … (excerpt)

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_f3683946
While common implementations of these algorithms employ L 2 regularization (often calling it “weight decay” in what may be misleading due to the inequivalence we expose), we propose a simple … (excerpt)

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ccbd6dee
We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam … (excerpt)

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_e233cd40
We show that a major factor of the poor generalization of the most popular adaptive gradient method, Adam, is due to the fact that L2 regularization is not nearly as … (excerpt)

I NTRODUCTION · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2af5de35
One possible explanation why Adam and other adaptive gradient methods might be outperformed by SGD with momentum is that common deep learning libraries only implement L 2 regularization, not the … (excerpt)

I NTRODUCTION · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_49bd6e03
In a comprehensive analysis, we show that Adam generalizes substantially better with decoupled weight decay than with L2 regularization, achieving 15% relative improvement in test error (see Figures 2 and … (excerpt)

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_b31b59b8
We demonstrate this formally for a simple special case of adaptive gradient algorithm with a fixed preconditioner: Proposition 3 (Weight decay = scale-adjusted L2 reg for adaptive gradient algorithm with … (excerpt)

D ECOUPLING THE WEIGHT DECAY FROM THE GRADIENT -BASED UPDATE · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d9b883ea
Our experimental setup follows that of Gastaldi (2017), who proposed, in addition to L 2 regularization, to apply the new Shake-Shake regularization to a 3-branch residual DNN that allowed to … (excerpt)

E XPERIMENTAL VALIDATION · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7fdfeb28
Since Adam already adapts its parameterwise learning rates it is not as common to use a learning rate multiplier schedule with it as it is with SGD, but as our … (excerpt)

E VALUATING DECOUPLED WEIGHT DECAY WITH DIFFERENT LEARNING RATE · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e2ffacfd
Figure 1 shows that decoupled weight decay outperforms L2 regularization for all learning rate schedules, with larger differences for better learning rate schedules.

E VALUATING DECOUPLED WEIGHT DECAY WITH DIFFERENT LEARNING RATE · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_2d426578
The figure also shows that cosine annealing clearly outperforms the other learning rate schedules; we thus used cosine annealing for the remainder of the experiments.

E VALUATING DECOUPLED WEIGHT DECAY WITH DIFFERENT LEARNING RATE · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9fdf3890
Consider, e.g., the setting at the top left black circle (α = 1/2,λ = 1/8∗ 0.001); only changing either α orλ by itself would worsen results, while changing both of … (excerpt)

D ECOUPLING THE WEIGHT DECAY AND INITIAL LEARNING RATE PARAMETERS · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a3fbaf86
In summary, the results in Figure 2 support our hypothesis that the weight decay and learning rate hyperparameters can be decoupled, and that this in turn simplifies the problem of … (excerpt)

D ECOUPLING THE WEIGHT DECAY AND INITIAL LEARNING RATE PARAMETERS · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c3747268
The results on ImageNet32x32 (see SuppFigure 4 in the Appendix) yield the same conclusion of substantially improved generalization performance.

B ETTER GENERALIZATION OF ADAMW · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b18cc462
For the default learning rate of 0.001, AdamW achieved 15% relative improvement in test error compared to Adam both on CIFAR-10 (also see SuppFigure 5) and ImageNet32x32 (also see SuppFigure … (excerpt)

A DAMWR WITH WARM RESTARTS FOR BETTER ANYTIME PERFORMANCE · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_34c5be19
These improvements closed most of the gap between Adam and SGDWR on CIFAR-10 and yielded comparable performance on ImageNet32x32.

A DAMWR WITH WARM RESTARTS FOR BETTER ANYTIME PERFORMANCE · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bb02c7a3
AdamW with the same hyperparameter setting as Adam yielded higher test set accuracy on Deep4Net (73.68% versus 71.37%) and statistically significantly higher test set accuracy on ResNet (72.04% versus 61.34%).

U SE OF ADAMW ON OTHER DATASETS AND ARCHITECTURES · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_69a7dea2
Here, we propose to reduce this dependence by normalizing the values of weight decay.

N ORMALIZED WEIGHT DECAY · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_eec2b9ca
SGDR has been successfully adopted to lead to new state-of-the-art results for popular image classification benchmarks (Huang et al., 2017; Gastaldi, 2017; Zoph et al., 2017), and we therefore already … (excerpt)

A DAM WITH COSINE ANNEALING AND WARM RESTARTS · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_554c2464
(15) In order to achieve good anytime performance, one can start with an initially small Ti (e.g., from 1% to 10% of the expected total budget) and multiply it by … (excerpt)

A DAM WITH COSINE ANNEALING AND WARM RESTARTS · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a4417327
These results are not very surprising given Figure 1 in the main paper (which demonstrates both the improvements possible by using some learning rate schedule, such as cosine annealing, and … (excerpt)

A DDITIONAL RESULTS · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b94e3bed
In contrast, had we used the same raw weight decay valuesλ for ImageNet32x32 as for CIFAR- 10 and for the same number of epochs, without the proposed normalization, λ would … (excerpt)

A DDITIONAL RESULTS · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6c2c7cc3
The results show that Adam and its variants with decoupled weight decay converge faster (in terms of training loss) on CIFAR-10 than the corresponding SGD variants (the difference for ImageNet32x32 … (excerpt)

A DDITIONAL RESULTS · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e054a80d
Interestingly, SuppFigure 5 and SuppFigure 6 show that the restart variants AdamWR and SGDWR also demonstrate better generalization than AdamW and SGDW, respectively.

A DDITIONAL RESULTS · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1711.05101) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.