What Gwern thinks about neural networks
Pseudonymous essayist. Writes at gwern.net on AI scaling, statistics, genetics and long-running internet research, with heavy citation and a habit of revising old essays in place rather than leaving them wrong.
Gwern did not write this page.
We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.
Filter neural networks · Newest first
Only Gwern on neural networks. See everyone on neural networks
5 dated positions, 2021, in their own words. Our reading of what Gwern has said — not written or endorsed by them.
-
Their wordsThe Master Synthesis: intelligence is execution of small simplicity-weighted programs, best discovered by search over smooth loss landscapes like that of highly-overparameterized differentiable networks containing lottery-ticket subnetworks which are ensembled/averaged over, approaching Bayes-optimal reasoning in the limit (as nearest-neighbors-like high dimensional interpolation / memorization gives way to algorithmic generalization / interpolation on a more abstract level)
↗May 2021 Gwern.net Newslettergwern.substack.com 2nd of 3 in this piece
- 2 months earlier
-
Our readingUnsupervised learning trained on unfiltered, internet-scale image data now genuinely works.
Their wordsanother reminder that unsupervised learning is really working!
↗March 2021 Gwern.net Newslettergwern.substack.com 1st of 3 in this piece
-
Their wordsSupervised learning from ImageNet is now obsolete for transfer learning, and ImageNet just a contaminated validation set
↗March 2021 Gwern.net Newslettergwern.substack.com 2nd of 3 in this piece
- 3 weeks earlier
-
Their wordsit is increasingly clear that brains really do do backprop in some sense
↗February 2021 Gwern.net Newslettergwern.substack.com 1st of 3 in this piece
- 5 weeks earlier
-
Their wordsthe mixture-of-experts approach, while scaling stably, starts showing its limits
↗Jan 2021 Gwern.net Newslettergwern.substack.com 1st of 3 in this piece