korrents

pruning

Cutting connections that are not doing the work — in a trained network, and in a developing brain.

FilterEveryone, all time15 from 7 people
Group All Technologists 1Scientists 1Writers & journalists 1
Who Everyone JFJonathan Frankle 5 MCMichael Carbin 5 ARArmin Ronacher 1 CHCasey Handmer 1 CRCharan Ranganath 1 GWGwern 1 TWTommy Wood 1
When All time Past 1 yearPast 3 yearsPast 5 years
  1. AR

    Armin Ronacher quoted

    Creator of Flask and Jinja

    Our readingMost agent harnesses need some pruning during context compaction anyway for cost reasons, which Jev might help with.

    Their wordsmost harnesses need some pruning on compaction anyways for cost reasons which Jev might help with

    ↗@mitsuhiko on Xx.com

  2. 2 months earlier
  3. TW

    Tommy Wood quoted

    Associate professor at the University of Washington

    Our readingNeuroplasticity is as much the removal of connections as the growth of new ones, so pruning is a form of learning rather than a form of decay.

    Their wordsA lot of people don't realize that particularly brain development, but then also the refining of functions in the brain through neuroplasticity includes the pruning or removal of signapses. Often we think about it's all about growth and new connections, but actually it also involves the refining or removal of of other connections that aren't needed.

    ↗Accelerate Learning & Increase Cognitive Capacity | Dr. Tommy Woodyoutube.com 1st of 26 in this recording

  4. 2 years earlier
  5. CR

    Charan Ranganath quoted

    Professor of psychology and neuroscience at the University of California

    Our readingThe point of a good memory is not to remember more but to remember better, because forgetting most of what you experience is what human brains are built to do.

    Their wordsWhat I say in the book is is that you don’t want to remember more, you want to remember better, which means focusing on the things that are important. That’s what our brains are designed to do.

    ↗Charan Ranganath: Human Memory, Imagination, Deja Vu, and False Memories | Lex Fridman Podcast #430youtube.com 4th of 20 in this recording

  6. 7 months earlier
  7. CH

    Casey Handmer quoted

    Physicist, formerly at NASA's Jet Propulsion Laboratory

    Our readingBatteries will replace transmission as the cheapest way to keep the lights on, and the grid will shrink rather than grow.

    energybatteries

    Their wordswe will see batteries aggressively displace transmission as the most cost-effective way to ensure continuity of electricity supply, leading to an eventual pruning of the grid and drastic reduction in the average distance that power travels between production and consumption, even as the average time that power is stored increases.

    ↗Radical Energy Abundancecaseyhandmer.wordpress.com 4th of 7 in this piece

  8. 2 years earlier
  9. GW

    Gwern quoted

    Pseudonymous essayist

    Our readingIntelligence is the execution of small, simplicity-weighted programs found by search over smooth loss landscapes in overparameterized networks, approaching Bayes-optimal reasoning as scale increases.

    simplicity

    Their wordsThe Master Synthesis: intelligence is execution of small simplicity-weighted programs, best discovered by search over smooth loss landscapes like that of highly-overparameterized differentiable networks containing lottery-ticket subnetworks which are ensembled/averaged over, approaching Bayes-optimal reasoning in the limit (as nearest-neighbors-like high dimensional interpolation / memorization gives way to algorithmic generalization / interpolation on a more abstract level)

    ↗May 2021 Gwern.net Newslettergwern.substack.com 2nd of 3 in this piece

  10. 3 years earlier
  11. JF

    Jonathan Frankle quoted

    Chief AI scientist at Databricks

    Our readingPruning cuts the parameters of a trained network by over 90% without costing accuracy, so inference gets smaller and faster for free.

    neural networks

    Their wordsNeural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy.

    ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org All 10 from this paper

  13. MC

    Michael Carbin quoted

    Associate professor at MIT

    Our readingPruning cuts the parameters of a trained network by over 90% without costing accuracy, so inference gets smaller and faster for free.

    neural networks

    Their wordsNeural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy.

    ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org All 10 from this paper

  15. JF

    Jonathan Frankle quoted

    Our readingThe saving pruning offers is only ever at inference, because the sparse architectures it produces are hard to train from the start.

    Their wordsHowever, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.

    ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org All 10 from this paper

  17. MC

    Michael Carbin quoted

    Our readingThe saving pruning offers is only ever at inference, because the sparse architectures it produces are hard to train from the start.

    Their wordsHowever, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.

    ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org All 10 from this paper

  19. JF

    Jonathan Frankle quoted

    Our readingPruning does not merely shrink a network; it uncovers subnetworks whose original initialization was what made them trainable at all.

    Their wordsWe find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively.

    ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org All 10 from this paper

  21. MC

    Michael Carbin quoted

    Our readingPruning does not merely shrink a network; it uncovers subnetworks whose original initialization was what made them trainable at all.

    Their wordsWe find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively.

    ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org All 10 from this paper

  23. JF

    Jonathan Frankle quoted

    Our readingA winning ticket wins on its initial weights: the connections it keeps started at values that happen to make training work.

    Their wordsThe winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.

    ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org All 10 from this paper

  25. MC

    Michael Carbin quoted

    Our readingA winning ticket wins on its initial weights: the connections it keeps started at values that happen to make training work.

    Their wordsThe winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.

    ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org All 10 from this paper

  27. JF

    Jonathan Frankle quoted

    Our readingA small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it.

    Their wordsAbove this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.

    ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org All 10 from this paper

  29. MC

    Michael Carbin quoted

    Our readingA small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it.

    Their wordsAbove this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.

    ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org All 10 from this paper