Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)

Jonathan Frankle, Michael Carbin · 9 Mar 2018 · arxiv.org ↗

10 korrents from this paper

In plain words

Big networks started at random hide smaller ones inside that, kept at those starting weights, train alone to the same accuracy just as fast. Finding them means networks under 10-20% the size can train from scratch to match or beat the original. It reports experiments plus a prune-and-reset method supporting this against the usual trouble training sparse nets from the start.

Our summary of the paper, not the authors' words — written to be readable without the field's vocabulary, from the stored copy of the paper and nothing else. Drafted with xai:grok-4.5 and checked by a person. The authors' own sentences are the quotes below.

Near this, by wording

Papers whose claims are worded most like this one's, found by the same hourly pass that draws the map. It is a measure of LANGUAGE, not of agreement or of citation: two papers can be near each other here and flatly contradict one another.

Jonathan Frankle, Michael Carbin did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

Our reading
Pruning cuts the parameters of a trained network by over 90% without costing accuracy, so inference gets smaller and faster for free.

Their wordsNeural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy.

Interesting
Our reading
Pruning cuts the parameters of a trained network by over 90% without costing accuracy, so inference gets smaller and faster for free.

Their wordsNeural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy.

Interesting
Our reading
The saving pruning offers is only ever at inference, because the sparse architectures it produces are hard to train from the start.

Their wordsHowever, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.

Interesting
Our reading
The saving pruning offers is only ever at inference, because the sparse architectures it produces are hard to train from the start.

Their wordsHowever, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.

Interesting
Our reading
Pruning does not merely shrink a network; it uncovers subnetworks whose original initialization was what made them trainable at all.

Their wordsWe find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively.

Interesting
Our reading
Pruning does not merely shrink a network; it uncovers subnetworks whose original initialization was what made them trainable at all.

Their wordsWe find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively.

Interesting
Our reading
A winning ticket wins on its initial weights: the connections it keeps started at values that happen to make training work.

Their wordsThe winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.

Interesting
Our reading
A winning ticket wins on its initial weights: the connections it keeps started at values that happen to make training work.

Their wordsThe winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.

Interesting
Our reading
A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it.

Their wordsAbove this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.

Interesting
Our reading
A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it.

Their wordsAbove this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.

Interesting