korrents

neural networks

The architecture almost all modern machine learning is built on, and what changes when it gets deeper.

What people on korrents have said about neural networks, newest first — 16 positions from 11 people.

FilterEveryone, all time
Who Everyone AKAndrej Karpathy 3 JSJian Sun 2 KHKaiming He 2 SAScott Alexander 2 AGAlexey Guzey 1 ARArmin Ronacher 1 DHDemis Hassabis 1 GMGary Marcus 1 JFJonathan Frankle 1 MCMichael Carbin 1 YBYoshua Bengio 1
When All time Past 1 yearPast 3 yearsPast 5 years
  1. SA

    Scott Alexander quoted

    Our readingLarge language models are grown rather than built, and nobody really knows how the resulting working AI functions.

    Their wordsLarge language models are “grown, not built”. Researchers run training data through a neural network. Eventually this creates a working AI; nobody really knows how.

    ↗God Help Us, Let’s Try To Learn About Mechanistic Interpretability Techniquesastralcodexten.com

    LLMs

    Interesting
  3. SA

    Scott Alexander quoted

    Our readingNeural networks do not have a one-to-one mapping between individual neurons and concepts.

    Their wordsresearchers started by seeking a 1:1 mapping between neurons and concepts, like a neuron that always fired when the AI was thinking about cats, but quickly learned that nothing like that existed.

    ↗God Help Us, Let’s Try To Learn About Mechanistic Interpretability Techniquesastralcodexten.com

    Interesting
  4. 6 months earlier
  5. AK

    Andrej Karpathy quoted

    Our readingAn agent can understand a maximally simplified explanation and still be unable to come up with it — that gap is what is left of the expert's job.

    Their wordsso for example, micro GPT, like I asked I tried to get an agent to write micro GPT. So, I told it like try to boil down the simplest things. Like try to boil down my um neural network training to the simplest thing and it can't do it.

    ↗Skill Issue: Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AIyoutube.com 20th of 20 in this recording

    Interesting
  6. 4 months earlier
  7. AK

    Andrej Karpathy quoted

    Our readingA verifiable task can be optimised by reinforcement learning until a neural network performs it extremely well.

    Their wordsIf a task/job is verifiable, then it is optimizable directly or via reinforcement learning, and a neural net can be trained to work extremely well.

    ↗Verifiabilitykarpathy.bearblog.dev

    reinforcement learning

    Interesting
  8. 4 weeks earlier
  9. AK

    Andrej Karpathy quoted

    Our readingThe knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out.

    Their wordsAnd actually, you don't actually need or want the knowledge. I actually think that's probably actually holding back the neural networks overall because it's actually like getting them to rely on the knowledge a little too much sometimes.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 6th of 30 in this recording

    scaling laws

    Interesting
  10. 3 months earlier
  11. DH

    Demis Hassabis quoted

    Our readingAnything nature shaped can be learned efficiently by a classical neural network, because evolutionary processes leave structure behind.

    Their wordsAnd I think the reason that's possible is that in nature, natural systems have structure because they were subject to evolutionary processes that shape them. And if that's true, then you can maybe learn what that structure is.

    ↗Demis Hassabis: Future of AI, Simulating Reality, Physics and Video Games | Lex Fridman Podcast #475youtube.com 1st of 20 in this recording

    Interesting
  12. 9 months earlier
  13. GM

    Gary Marcus quoted

    Our readingScaling the 2019-2023 recipe — same architecture, bigger model, more data — is not enough; further progress depends on new architectural ideas.

    Their wordsNeurosymbolic AI — combining such machinery with neural networks – is likely a necessary condition for going forward.

    ↗LLMs don’t do formal reasoning - and that is a HUGE problemgarymarcus.substack.com 2nd of 2 in this piece

    Interesting
  14. 2 months earlier
  15. AG

    Alexey Guzey quoted

    Our readingWork on large language models is not progress toward artificial general intelligence, because the models do not understand what they read.

    Their wordsThis means that working on neural networks is NOT getting us closer to AGI, except indirectly.

    ↗How my views on AI changed every year 2017-2024guzey.com

    AGILLMs

    Interesting
  16. 14 months earlier
  17. YB

    Yoshua Bengio quoted

    Our readingAccess to large-scale generalist AI systems that could be weaponized should be limited, meaning their code and neural network parameters should not be released open-source.

    Their wordsAt the same time, in order to reduce the probability of someone intentionally or unintentionally bringing about a rogue AI, we need to increase governance and we should consider limiting access to the large-scale generalist AI systems that could be weaponized, which would mean that the code and neural net parameters would not be shared in open-source and some of the important engineering tricks to make them work would not be shared either.

    ↗FAQ on Catastrophic AI Risksyoshuabengio.org

    Interesting
  18. 4 months earlier
  19. AR

    Armin Ronacher quoted

    Our readingWe do not control neural networks well enough to guarantee an AI will not harm humans, so we should be careful about what capabilities we give them.

    Their wordsThe biggest issue is probably that we don’t control neutral networks enough to be able to ensure AI doesn’t harm humans. We can’t even control AI to not reveal internal prompts.

    ↗I Think AI Would Kill my Wifelucumr.pocoo.org

    Interesting
  20. 5 years earlier
  21. JF

    Jonathan Frankle quoted

    Our readingPruning cuts the parameters of a trained network by over 90% without costing accuracy, so inference gets smaller and faster for free.

    Their wordsNeural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy.

    ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org 1st of 10 in this piece

    pruning

    Interesting
  23. MC

    Michael Carbin quoted

    Our readingPruning cuts the parameters of a trained network by over 90% without costing accuracy, so inference gets smaller and faster for free.

    Their wordsNeural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy.

    ↗Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv)arxiv.org 2nd of 10 in this piece

    pruning

    Interesting
  24. 2 years earlier
  25. KH

    Kaiming He quoted

    Our readingDepth is not free: past a point, deeper neural networks are harder to train rather than simply better.

    Their wordsDeeper neural networks are more difficult to train.

    ↗He, Zhang, Ren & Sun, "Deep Residual Learning for Image Recognition" (arXiv)arxiv.org 1st of 8 in this piece

    Interesting
  27. JS

    Jian Sun quoted

    Our readingDepth is not free: past a point, deeper neural networks are harder to train rather than simply better.

    Their wordsDeeper neural networks are more difficult to train.

    ↗He, Zhang, Ren & Sun, "Deep Residual Learning for Image Recognition" (arXiv)arxiv.org 2nd of 8 in this piece

    Interesting
  29. KH

    Kaiming He quoted

    Our readingResidual networks are easier to optimize than plain ones and keep gaining accuracy as depth increases.

    Their wordsWe provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.

    ↗He, Zhang, Ren & Sun, "Deep Residual Learning for Image Recognition" (arXiv)arxiv.org 5th of 8 in this piece

    Interesting
  31. JS

    Jian Sun quoted

    Our readingResidual networks are easier to optimize than plain ones and keep gaining accuracy as depth increases.

    Their wordsWe provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.

    ↗He, Zhang, Ren & Sun, "Deep Residual Learning for Image Recognition" (arXiv)arxiv.org 6th of 8 in this piece

    Interesting