clm_9e7daaf5In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks.
arXiv 1608.03983 · cs.LG cs.NE math.OC
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_9e7daaf5In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks.
res_55b0f837We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively.
clm_69ea52e2In this paper, we propose to periodically simulate warm restarts of SGD, where in each restart the learning rate is initialized to some value and is scheduled to decrease.
res_6603a831Our empirical results suggest that SGD with warm restarts requires 2× to 4× fewer epochs than the currently-used learning rate schedule schemes to achieve comparable or even better results.
res_b2052061Furthermore, combining the networks obtained right before restarts in an ensemble following the approach proposed by Huang et al. (2016a) improves our results further to 3.14% for CIFAR-10 and 16.21% … (excerpt)
res_68d1cb5aIn order to improve anytime performance, we suggest an option to start with an initially small Ti and increase it by a factor of Tmult at every restart (see, e.g., … (excerpt)
res_41105be8The top and middle rows show the same results on WRN-28-10, with the middle row zooming into the good performance region of low test error.
res_baea1ab4The results for η0 = 0.05 show better performance, and therefore we useη0 = 0.05 in our later experiments.
res_3ecaac75We therefore investigated whether results on CIFAR-10 and CIFAR-100 can be further improved by making WRNs two times wider, i.e., by training WRN-28-20 instead of WRN-28-10.
res_7504570aTable 1 shows that the results indeed improved, by about 0.25% on CIFAR-10 and by about 0.5-1.0% on CIFAR-100.
res_b58d1d16While network architecture WRN-28-20 requires roughly three-four times more computation than WRN-28-10, the aggressive learning rate reduction of SGDR nevertheless allowed us to achieve a better error rate in the … (excerpt)
res_c911736bSpecifically, Figure 2 (right middle and right bottom) show that after only 50 epochs, SGDR (even without restarts, usingT0 = 50,T mult = 1) achieved an error rate below 19% … (excerpt)
clm_d3f6a5d4We therefore have hope that – by enabling researchers to test new architectures faster – SGDR’s good anytime performance may also lead to improvements of the state of the art.
res_d4f148cdAs the figure shows, SGDR optimizes training loss faster than the standard default schedule until about epoch 120.
res_c4ea286fThe original test error of 4.03% on CIFAR-10 and 19.57% on CIFAR-100 (median of 16 runs) can be improved to 3.51% on CIFAR-10 and 17.75% on CIFAR-100 whenM = 3 … (excerpt)
res_244f7acfTo achieve the same result, one would have to aggregateN = 3 models obtained at epoch 150 ofN = 3 independent runs (seeN = 3,M = 1 in Figure 3).
res_c882eb19We believe that these results could be further improved by considering better baseline models than WRN-28-10 (e.g., WRN-28-20).
res_fdbc4366SGDR with T0 = 10,T mult = 2,η imax = 0.01 achieves top-1 error of 39.24% and top-5 error of 17.17% matching the original results by AlexNets (40.7% and 18.2%, … (excerpt)
res_6f52293fClearly, longer runs (more than 40 epochs considered in this preliminary experiment) and hyperparameter tuning of learning rates, regularization and other hyperparameters shall further improve the results.
clm_d1b499edOur results suggest that even without any restarts the proposed aggressive learning rate schedule given by eq.
res_a1523d52Our SGDR simulates warm restarts by scheduling the learning rate to achieve competitive results on CIFAR-10 and CIFAR-100 roughly two to four times faster.
res_8093feb8We also achieved new stateof-the-art results with SGDR, mainly by using even wider WRNs and ensembles of snapshots from 10 Published as a conference paper at ICLR 2017 SGDR’s trajectory.
res_654bef47In contrast to the experiments described in the main paper, here, the dataset is permuted only within 10 subgroups each formed from 100 classes which makes good generalization much harder … (excerpt)
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1608.03983) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.