res_2e06e022Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy.
arXiv 1803.03635 · cs.LG cs.AI cs.NE
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
res_2e06e022Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy.
res_a1c158a8We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10.
clm_99c259eeIn this paper, we show that there consistently exist smaller subnetworks that train from the start and learn at least as fast as their larger counterparts while reaching similar test … (excerpt)
res_bae5c367The lottery ticket hypothesis predicts that∃m for whichj′≤j (commensurate training time),a′≥a (commensurate accuracy), and‖m‖0≪|θ| (fewer parameters).
clm_6151d8b2Our results show that iterative pruning finds winning tickets that match the accuracy of the original network at smaller sizes than does one-shot pruning.
clm_ea1bad7cContributions. • We demonstrate that pruning uncovers trainable subnetworks that reach test accuracy comparable to the original networks from which they derived in a comparable number of iterations. • We … (excerpt)
res_e6a4a0ecFor the first pruning rounds, networks learn faster and reach higher test accuracy the more they are pruned (left graph in Figure 3).
res_4512cd91A winning ticket comprising 51.3% of the weights from the original network (i.e.,Pm = 51.3%) reaches higher test accuracy faster than the original network but slower than whenPm = 21.1%.
res_1769e261The winning tickets learn faster asPm decreases from 100% to 21%, at which point early-stopping occurs 38% earlier than for the original network.
res_bc2951cdFurther pruning causes learning to slow, returning to the early-stopping performance of the original network whenPm = 3.6%.
res_28fc0e88Test accuracy increases with pruning, improving by more than 0.3 percentage points when Pm = 13.5%; after this point, accuracy decreases, returning to the level of the original network whenPm … (excerpt)
res_e4cda5d4At early stopping, training accuracy (Figure 4a, right) increases with pruning in a similar pattern to test accuracy, seemingly implying that winning tickets optimize more effectively but do not generalize … (excerpt)
res_58ffceebHowever, at iteration 50,000 (Figure 4b), iteratively-pruned winning tickets still see a test accuracy improvement of up to 0.35 percentage points in spite of the fact that training accuracy reaches … (excerpt)
res_e3860e08WhenPm = 21%, the winning ticket reaches minimum validation loss 2.51x faster than when reinitialized and is half a percentage point more accurate.
res_af408ee3When 95.0%≥Pm≥ 5.17%, test accuracy is higher than the original network.
res_6e3fbdf4The pattern from Lenet in Section 2 repeats: as the network is pruned, it learns faster and test accuracy rises as compared to the original network.
res_0eb8a63fThe bottom right graph plots test accuracy of winning tickets at iterations corresponding to the last iteration of training for the original network (20,000 for Conv-2, 25,000 for Conv-4, and … (excerpt)
res_85bb9218However, at iteration 20,000 for Conv-2, 25,000 for Conv-4, and 30,000 for Conv-6 (the iterations corresponding to the final training iteration for the original network), training accuracy reaches 100% for … (excerpt)
res_21758a6dHowever, unlike Lenet, test accuracy at early-stopping time initially remains steady and even improves for Conv-2 and Conv-4, indicating that—at moderate levels of pruning—the structure of the winning tickets alone … (excerpt)
res_9edf4c12However, at the lower learning rate, the usual pattern reemerges, with subnetworks that remain within 1 percentage point of the original accuracy whilePm≥ 3.5%.
res_1cdc1757Although these subnetworks learn faster than the unpruned network early in training (Figure 7 left), this accuracy advantage erodes later in training due to the lower initial learning rate.
res_06437c72To bridge the gap between the lottery ticket behavior of the lower learning rate and the accuracy advantage of the higher learning rate, we explore the effect of linear learning … (excerpt)
res_5593637fTraining VGG-19 with warmup (k = 10000, green line) at learning rate 0.1 improves the test accuracy of the unpruned network by about one percentage point.
res_4dd50827Warmup makes it possible to find winning tickets, exceeding this initial accuracy whenPm≥ 1.5%.
res_9dcf5f98The accuracy of the best winning tickets at the lower learning rate (89.5% when 41.7%≥ Pm≥ 21.9%) falls short of the original network’s accuracy at the higher learning rate (90.5%).
res_4f28deffWinning tickets trained with warmup close the accuracy gap with the unpruned network at the higher learning rate, reaching 90.5% test accuracy with learning rate 0.03 (warmup,k = 20000) atPm … (excerpt)
res_4bdada46However, after further pruning, initialization matters: we find winning tickets when VGG-19 is pruned by up to 98.5%; when reinitialized, these tickets reach much lower accuracy.
clm_855218e4We show that dense networks contain sparse subnetworks capable of learning on their own starting from their original initializations.
clm_373bb1edWe show that one need not even update all parameters to optimize a network, and we find winning tickets through a principled search process involving pruning.
res_489aae7dIn all cases, the Strategy 1 maintains higher validation accuracy and faster early-stopping times to smaller network sizes.
res_8d86f2b0We consider networks that reach this moment sooner to have learned “faster.” In support of this notion, the ordering in which each experiment meets our early-stopping criterion in Figure 3 … (excerpt)
res_31adfa2aAs we describe in the main body of the paper, training accuracy reaches 100% in all cases for all but the most heavily pruned networks.
clm_6f0ef191For the fully-connected Lenet architecture for MNIST, we find that the randomly reinitialized networks outperform random sparsity.
res_c8b7b101Even after adding noise of 3σ, the winning tickets continue to outperform the random reinitialization experiment.
res_a64dc015According to the graph on the right of Figure 26, several learning rates between 0.0002 and 0.002 achieve similar levels of validation accuracy on the original network and maintain that … (excerpt)
res_23053f0fWe choose 0.0012 due to its higher validation accuracy on the unpruned network and in consideration of criterion (3) above.
res_e110e1b4Even for those learning rates that did not satisfy the early-stopping criterion within 50,000 iterations (2.5e-05 and 0.0064) still showed accuracy improvements with pruning.
res_ccc1e137Once again, the lottery ticket pattern appears across all learning rates, with learning rates between 0.025 and 0.1 maintaining high validation accuracy and faster learning for the longest number of … (excerpt)
res_d8e60ce1Learning rate 0.025 achieves the highest validation accuracy on the unpruned network; however, its validation accuracy never increases as it is pruned, instead decreasing gradually, and higher learning rates reach … (excerpt)
res_3c76a671There is a tangible difference in learning speed and validation accuracy at early-stopping between the lowest pruning rates (0.1 and 0.2) and higher pruning rates (0.4 and above).
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1803.03635) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.