korrents
Gwern

Gwern

@gwern · 15 positions · On ppll ↗

Pseudonymous essayist. Writes at gwern.net on AI scaling, statistics, genetics and long-running internet research, with heavy citation and a habit of revising old essays in place rather than leaving them wrong.

Gwern did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

  1. benchmarkssimplicity

    The more people tweet about how MLP-Mixer is overhyped because is −X% worse than the ultra-hand-optimized baseline or requires Y× more FLOPS, the more they demonstrate precisely why this sort of research is so important!

    ↗May 2021 Gwern.net Newslettergwern.substack.com 1st of 3 in this piece

  2. AGIsimplicityneural networkspruning

    The Master Synthesis: intelligence is execution of small simplicity-weighted programs, best discovered by search over smooth loss landscapes like that of highly-overparameterized differentiable networks containing lottery-ticket subnetworks which are ensembled/averaged over, approaching Bayes-optimal reasoning in the limit (as nearest-neighbors-like high dimensional interpolation / memorization gives way to algorithmic generalization / interpolation on a more abstract level)

    ↗May 2021 Gwern.net Newslettergwern.substack.com 2nd of 3 in this piece

  3. measuring intelligence

    human intelligence's g is real but is the overall 'pool' of neural resources which derives from overall body integrity because the number of neurons, their density, their myelination, resistance to damage and infection etc, is causally downstream of all body and developmental systems, creating a huge mutational target

    ↗May 2021 Gwern.net Newslettergwern.substack.com 3rd of 3 in this piece

  4. 8 days earlier
  5. transformersLLMsinnovation

    if the marginal return on this research direction had fallen below that of competitors, we would see those neglected directions invade Transformer topics-while we continue to see the reverse, and many applications as yet untouched by all the new approaches, suggesting that we still don't pay enough attention

    ↗April 2021 newslettergwern.substack.com 1st of 3 in this piece

  6. metascience

    there's nothing so expensive as demanding research be cheap.

    ↗April 2021 newslettergwern.substack.com 2nd of 3 in this piece

  7. nothing in psychology, or medicine, makes sense except in the light of individual differences

    ↗April 2021 newslettergwern.substack.com 3rd of 3 in this piece

  8. 8 weeks earlier
  9. neural networkscomputer vision

    another reminder that unsupervised learning is really working!

    ↗March 2021 Gwern.net Newslettergwern.substack.com 1st of 3 in this piece

  10. neural networks

    Supervised learning from ImageNet is now obsolete for transfer learning, and ImageNet just a contaminated validation set

    ↗March 2021 Gwern.net Newslettergwern.substack.com 2nd of 3 in this piece

  11. health and fitness

    a self-blinding study, similar to my old self-blinding protocols, confirms that microdosing is just placebo effect, as I said in 2012

    ↗March 2021 Gwern.net Newslettergwern.substack.com 3rd of 3 in this piece

  12. 3 weeks earlier
  13. neural networks

    it is increasingly clear that brains really do do backprop in some sense

    ↗February 2021 Gwern.net Newslettergwern.substack.com 1st of 3 in this piece

  14. governmentspace

    NASA's lunar quarantine was a sham intended to mollify the public as they covered up repeated major failures & lab leaks both before & after-had there been any dangerous lunar organisms, they would have escaped easily

    ↗February 2021 Gwern.net Newslettergwern.substack.com 2nd of 3 in this piece

  15. inflationinvesting

    it's not one thing, it's everything-more intelligent people have lower discount rates, save more for longer & are less risk-averse, more accurately predict future growth or inflation

    ↗February 2021 Gwern.net Newslettergwern.substack.com 3rd of 3 in this piece

  16. 5 weeks earlier
  17. LLMsscaling lawsneural networks

    the mixture-of-experts approach, while scaling stably, starts showing its limits

    ↗Jan 2021 Gwern.net Newslettergwern.substack.com 1st of 3 in this piece

  18. GANs

    To take the GAN/DRL analogy seriously, perhaps they were they ultimately a dead end, akin to trying to learn everything from rewards, and an adversarial GAN loss ought to be only the cherry on the cake of a large unsupervised/semi-supervised generative model.

    ↗Jan 2021 Gwern.net Newslettergwern.substack.com 2nd of 3 in this piece

  19. transformersLLMs

    The fact that the prefix-tuning, by directly optimizing the prompt embeddings, yields better performance than even single optimized text prompts, suggests so.

    ↗Jan 2021 Gwern.net Newslettergwern.substack.com 3rd of 3 in this piece