MearcAll papers

arXiv 1608.03983 · cs.LG cs.NE math.OC

SGDR: Stochastic Gradient Descent with Warm Restarts

Ilya Loshchilov, Frank Hutter

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_9e7daaf5
In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_55b0f837
We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_69ea52e2
In this paper, we propose to periodically simulate warm restarts of SGD, where in each restart the learning rate is initialized to some value and is scheduled to decrease.

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6603a831
Our empirical results suggest that SGD with warm restarts requires 2× to 4× fewer epochs than the currently-used learning rate schedule schemes to achieve comparable or even better results.

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b2052061
Furthermore, combining the networks obtained right before restarts in an ensemble following the approach proposed by Huang et al. (2016a) improves our results further to 3.14% for CIFAR-10 and 16.21% … (excerpt)

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_68d1cb5a
In order to improve anytime performance, we suggest an option to start with an initially small Ti and increase it by a factor of Tmult at every restart (see, e.g., … (excerpt)

S TOCHASTIC GRADIENT DESCENT WITH WARM RESTARTS (SGDR) · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_41105be8
The top and middle rows show the same results on WRN-28-10, with the middle row zooming into the good performance region of low test error.

E XPERIMENTAL SETTINGS · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_baea1ab4
The results for η0 = 0.05 show better performance, and therefore we useη0 = 0.05 in our later experiments.

S INGLE -M ODEL RESULTS · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3ecaac75
We therefore investigated whether results on CIFAR-10 and CIFAR-100 can be further improved by making WRNs two times wider, i.e., by training WRN-28-20 instead of WRN-28-10.

S INGLE -M ODEL RESULTS · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7504570a
Table 1 shows that the results indeed improved, by about 0.25% on CIFAR-10 and by about 0.5-1.0% on CIFAR-100.

S INGLE -M ODEL RESULTS · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b58d1d16
While network architecture WRN-28-20 requires roughly three-four times more computation than WRN-28-10, the aggressive learning rate reduction of SGDR nevertheless allowed us to achieve a better error rate in the … (excerpt)

S INGLE -M ODEL RESULTS · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c911736b
Specifically, Figure 2 (right middle and right bottom) show that after only 50 epochs, SGDR (even without restarts, usingT0 = 50,T mult = 1) achieved an error rate below 19% … (excerpt)

S INGLE -M ODEL RESULTS · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_d3f6a5d4
We therefore have hope that – by enabling researchers to test new architectures faster – SGDR’s good anytime performance may also lead to improvements of the state of the art.

S INGLE -M ODEL RESULTS · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d4f148cd
As the figure shows, SGDR optimizes training loss faster than the standard default schedule until about epoch 120.

S INGLE -M ODEL RESULTS · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c4ea286f
The original test error of 4.03% on CIFAR-10 and 19.57% on CIFAR-100 (median of 16 runs) can be improved to 3.51% on CIFAR-10 and 17.75% on CIFAR-100 whenM = 3 … (excerpt)

E NSEMBLE RESULTS · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_244f7acf
To achieve the same result, one would have to aggregateN = 3 models obtained at epoch 150 ofN = 3 independent runs (seeN = 3,M = 1 in Figure 3).

E NSEMBLE RESULTS · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c882eb19
We believe that these results could be further improved by considering better baseline models than WRN-28-10 (e.g., WRN-28-20).

E NSEMBLE RESULTS · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fdbc4366
SGDR with T0 = 10,T mult = 2,η imax = 0.01 achieves top-1 error of 39.24% and top-5 error of 17.17% matching the original results by AlexNets (40.7% and 18.2%, … (excerpt)

P RELIMINARY EXPERIMENTS ON A DOWNSAMPLED IMAGE NET DATASET · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6f52293f
Clearly, longer runs (more than 40 epochs considered in this preliminary experiment) and hyperparameter tuning of learning rates, regularization and other hyperparameters shall further improve the results.

P RELIMINARY EXPERIMENTS ON A DOWNSAMPLED IMAGE NET DATASET · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_d1b499ed
Our results suggest that even without any restarts the proposed aggressive learning rate schedule given by eq.

D ISCUSSION · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a1523d52
Our SGDR simulates warm restarts by scheduling the learning rate to achieve competitive results on CIFAR-10 and CIFAR-100 roughly two to four times faster.

C ONCLUSION · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8093feb8
We also achieved new stateof-the-art results with SGDR, mainly by using even wider WRNs and ensembles of snapshots from 10 Published as a conference paper at ICLR 2017 SGDR’s trajectory.

C ONCLUSION · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_654bef47
In contrast to the experiments described in the main paper, here, the dataset is permuted only within 10 subgroups each formed from 100 classes which makes good generalization much harder … (excerpt)

S UPPLEMENTARY MATERIAL · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1608.03983) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.