What Jonathan Frankle thinks about pruning

@jonathan-frankle · 5 positions · On ppll ↗

Chief AI scientist at Databricks, and the author of the lottery ticket hypothesis. His MIT doctorate was on why the sparse networks pruning produces are so hard to train from scratch.

Jonathan Frankle did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

Filter pruning · Newest first

Topic

All 5pruning 5neural networks 4

Sort

NewestOldest

Only Jonathan Frankle on pruning. See everyone on pruning

5 dated positions, 2018, in their own words. Our reading of what Jonathan Frankle has said — not written or endorsed by them.

1. 9 Mar 2018

   Our readingPruning cuts the parameters of a trained network by over 90% without costing accuracy, so inference gets smaller and faster for free.

   neural networks

   Their wordsNeural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy.

   ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org All 10 from this paper

3. 9 Mar 2018

   Our readingThe saving pruning offers is only ever at inference, because the sparse architectures it produces are hard to train from the start.

   Their wordsHowever, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.

   ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org All 10 from this paper

5. 9 Mar 2018

   Our readingPruning does not merely shrink a network; it uncovers subnetworks whose original initialization was what made them trainable at all.

   Their wordsWe find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively.

   ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org All 10 from this paper

7. 9 Mar 2018

   Our readingA winning ticket wins on its initial weights: the connections it keeps started at values that happen to make training work.

   Their wordsThe winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.

   ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org All 10 from this paper

9. 9 Mar 2018

   Our readingA small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it.

   Their wordsAbove this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.

   ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org All 10 from this paper