MearcAll papers

arXiv 1803.03635 · cs.LG cs.AI cs.NE

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Jonathan Frankle, Michael Carbin

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedres_2e06e022
Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a1c158a8
We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_99c259ee
In this paper, we show that there consistently exist smaller subnetworks that train from the start and learn at least as fast as their larger counterparts while reaching similar test … (excerpt)

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bae5c367
The lottery ticket hypothesis predicts that∃m for whichj′≤j (commensurate training time),a′≥a (commensurate accuracy), and‖m‖0≪|θ| (fewer parameters).

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_6151d8b2
Our results show that iterative pruning finds winning tickets that match the accuracy of the original network at smaller sizes than does one-shot pruning.

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ea1bad7c
Contributions. • We demonstrate that pruning uncovers trainable subnetworks that reach test accuracy comparable to the original networks from which they derived in a comparable number of iterations. • We … (excerpt)

I NTRODUCTION · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e6a4a0ec
For the first pruning rounds, networks learn faster and reach higher test accuracy the more they are pruned (left graph in Figure 3).

W INNING TICKETS IN FULLY-CONNECTED NETWORKS · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4512cd91
A winning ticket comprising 51.3% of the weights from the original network (i.e.,Pm = 51.3%) reaches higher test accuracy faster than the original network but slower than whenPm = 21.1%.

W INNING TICKETS IN FULLY-CONNECTED NETWORKS · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1769e261
The winning tickets learn faster asPm decreases from 100% to 21%, at which point early-stopping occurs 38% earlier than for the original network.

W INNING TICKETS IN FULLY-CONNECTED NETWORKS · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bc2951cd
Further pruning causes learning to slow, returning to the early-stopping performance of the original network whenPm = 3.6%.

W INNING TICKETS IN FULLY-CONNECTED NETWORKS · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_28fc0e88
Test accuracy increases with pruning, improving by more than 0.3 percentage points when Pm = 13.5%; after this point, accuracy decreases, returning to the level of the original network whenPm … (excerpt)

W INNING TICKETS IN FULLY-CONNECTED NETWORKS · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e4cda5d4
At early stopping, training accuracy (Figure 4a, right) increases with pruning in a similar pattern to test accuracy, seemingly implying that winning tickets optimize more effectively but do not generalize … (excerpt)

W INNING TICKETS IN FULLY-CONNECTED NETWORKS · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_58ffceeb
However, at iteration 50,000 (Figure 4b), iteratively-pruned winning tickets still see a test accuracy improvement of up to 0.35 percentage points in spite of the fact that training accuracy reaches … (excerpt)

W INNING TICKETS IN FULLY-CONNECTED NETWORKS · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e3860e08
WhenPm = 21%, the winning ticket reaches minimum validation loss 2.51x faster than when reinitialized and is half a percentage point more accurate.

W INNING TICKETS IN FULLY-CONNECTED NETWORKS · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_af408ee3
When 95.0%≥Pm≥ 5.17%, test accuracy is higher than the original network.

W INNING TICKETS IN FULLY-CONNECTED NETWORKS · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6e3fbdf4
The pattern from Lenet in Section 2 repeats: as the network is pruned, it learns faster and test accuracy rises as compared to the original network.

W INNING TICKETS IN CONVOLUTIONAL NETWORKS · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0eb8a63f
The bottom right graph plots test accuracy of winning tickets at iterations corresponding to the last iteration of training for the original network (20,000 for Conv-2, 25,000 for Conv-4, and … (excerpt)

W INNING TICKETS IN CONVOLUTIONAL NETWORKS · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_85bb9218
However, at iteration 20,000 for Conv-2, 25,000 for Conv-4, and 30,000 for Conv-6 (the iterations corresponding to the final training iteration for the original network), training accuracy reaches 100% for … (excerpt)

W INNING TICKETS IN CONVOLUTIONAL NETWORKS · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_21758a6d
However, unlike Lenet, test accuracy at early-stopping time initially remains steady and even improves for Conv-2 and Conv-4, indicating that—at moderate levels of pruning—the structure of the winning tickets alone … (excerpt)

W INNING TICKETS IN CONVOLUTIONAL NETWORKS · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9edf4c12
However, at the lower learning rate, the usual pattern reemerges, with subnetworks that remain within 1 percentage point of the original accuracy whilePm≥ 3.5%.

VGG AND RESNET FOR CIFAR10 · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1cdc1757
Although these subnetworks learn faster than the unpruned network early in training (Figure 7 left), this accuracy advantage erodes later in training due to the lower initial learning rate.

VGG AND RESNET FOR CIFAR10 · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_06437c72
To bridge the gap between the lottery ticket behavior of the lower learning rate and the accuracy advantage of the higher learning rate, we explore the effect of linear learning … (excerpt)

VGG AND RESNET FOR CIFAR10 · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5593637f
Training VGG-19 with warmup (k = 10000, green line) at learning rate 0.1 improves the test accuracy of the unpruned network by about one percentage point.

VGG AND RESNET FOR CIFAR10 · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4dd50827
Warmup makes it possible to find winning tickets, exceeding this initial accuracy whenPm≥ 1.5%.

VGG AND RESNET FOR CIFAR10 · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9dcf5f98
The accuracy of the best winning tickets at the lower learning rate (89.5% when 41.7%≥ Pm≥ 21.9%) falls short of the original network’s accuracy at the higher learning rate (90.5%).

VGG AND RESNET FOR CIFAR10 · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4f28deff
Winning tickets trained with warmup close the accuracy gap with the unpruned network at the higher learning rate, reaching 90.5% test accuracy with learning rate 0.03 (warmup,k = 20000) atPm … (excerpt)

VGG AND RESNET FOR CIFAR10 · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4bdada46
However, after further pruning, initialization matters: we find winning tickets when VGG-19 is pruned by up to 98.5%; when reinitialized, these tickets reach much lower accuracy.

D ISCUSSION · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_855218e4
We show that dense networks contain sparse subnetworks capable of learning on their own starting from their original initializations.

R ELATED WORK · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_373bb1ed
We show that one need not even update all parameters to optimize a network, and we find winning tickets through a principled search process involving pruning.

R ELATED WORK · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_489aae7d
In all cases, the Strategy 1 maintains higher validation accuracy and faster early-stopping times to smaller network sizes.

I TERATIVE PRUNING STRATEGIES · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8d86f2b0
We consider networks that reach this moment sooner to have learned “faster.” In support of this notion, the ordering in which each experiment meets our early-stopping criterion in Figure 3 … (excerpt)

E ARLY STOPPING CRITERION · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_31adfa2a
As we describe in the main body of the paper, training accuracy reaches 100% in all cases for all but the most heavily pruned networks.

T RAINING ACCURACY FOR LOTTERY TICKET EXPERIMENTS · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_6f0ef191
For the fully-connected Lenet architecture for MNIST, we find that the randomly reinitialized networks outperform random sparsity.

C OMPARING RANDOM REINITIALIZATION AND RANDOM SPARSITY · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c8b7b101
Even after adding noise of 3σ, the winning tickets continue to outperform the random reinitialization experiment.

A DDING NOISE TO WINNING TICKETS · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a64dc015
According to the graph on the right of Figure 26, several learning rates between 0.0002 and 0.002 achieve similar levels of validation accuracy on the original network and maintain that … (excerpt)

L EARNING RATE · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_23053f0f
We choose 0.0012 due to its higher validation accuracy on the unpruned network and in consideration of criterion (3) above.

L EARNING RATE · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e110e1b4
Even for those learning rates that did not satisfy the early-stopping criterion within 50,000 iterations (2.5e-05 and 0.0064) still showed accuracy improvements with pruning.

L EARNING RATE · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ccc1e137
Once again, the lottery ticket pattern appears across all learning rates, with learning rates between 0.025 and 0.1 maintaining high validation accuracy and faster learning for the longest number of … (excerpt)

M OMENTUM · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d8e60ce1
Learning rate 0.025 achieves the highest validation accuracy on the unpruned network; however, its validation accuracy never increases as it is pruned, instead decreasing gradually, and higher learning rates reach … (excerpt)

M OMENTUM · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3c76a671
There is a tangible difference in learning speed and validation accuracy at early-stopping between the lowest pruning rates (0.1 and 0.2) and higher pruning rates (0.4 and above).

I TERATIVE PRUNING RATE · page 27 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1803.03635) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.