korrents

neural networks

The architecture almost all modern machine learning is built on, and what changes when it gets deeper.

FilterEveryone, all time87 from 44 people
Group All Technologists 4Scientists 6Artists & entertainers 1Writers & journalists 3
Who Everyone YBYoshua Bengio 6 GWGwern 5 AKAndrej Karpathy 4 JSJian Sun 4 JFJonathan Frankle 4 KHKaiming He 4 MCMichael Carbin 4 COChristopher Olah 3 AGAidan N. Gomez 2 AVAshish Vaswani 2 IPIllia Polosukhin 2 JDJacob Devlin 2 JUJakob Uszkoreit 2 JSJürgen Schmidhuber 2 KLKenton Lee 2 KTKristina Toutanova 2 LWLilian Weng 2 LJLlion Jones 2 ŁKŁukasz Kaiser 2 MCMing-Wei Chang 2 NPNiki Parmar 2 NSNoam Shazeer 2 RSRichard Sutton 2 SAScott Alexander 2 SHSepp Hochreiter 2 AWAdam Wiggins 1 ATAlex Telford 1 AGAlexey Guzey 1 ASAravind Srinivas 1 ARArmin Ronacher 1 BCBoris Cherny 1 BDBrad DeLong 1 CHChip Huyen 1 DBDean W. Ball 1 DHDemis Hassabis 1 EYEliezer Yudkowsky 1 GMGary Marcus 1 KKKevin Kelly 1 NLNathan Lambert 1 NNNeel Nanda 1 NENelson Elhage 1 NSNoah Smith 1 SRSebastian Raschka 1 SRSebastian Ruder 1
When All time Past 1 yearPast 3 yearsPast 5 years
  1. DB

    Dean W. Ball quoted

    Head of strategic futures at OpenAI

    Our readingIt may be the case that neural networks will never be perfectly explainable in a way that satisfies human rationality.

    Their wordsIt may well be the case that neural networks will never be perfectly explainable, at least not in a way that satisfies human rationality.

    ↗The Beauty of Something Grownhyperdimensional.co

  2. 1 day earlier
  3. NS

    Noah Smith quoted

    Economics writer; author of the Noahpinion newsletter

    Our readingNeural networks can be inspected to understand how they work, unlike human brains.

    Their wordsUnlike human brains, we can look inside neutral networks and figure out how they work!

    ↗@Noahpinion on Xx.com

  4. 3 weeks earlier
  5. BD

    Brad DeLong quoted

    Economics professor at the University of California

    Our readingAI is unlikely to be a hyperprofits engine for research labs whose attached companies make money serving neural-network calculations from datacenters.

    data centers

    Their wordsThis week brought me news increasing my confidence with respect to one thing it is not: it is unlikely to be a hyperprofits engine for research labs with companies attached making money serving up calculations made in datacenters by simulated neural networks.

    ↗Real Technology, Phantom Profits: Who Is Likely to Actually Get Paid for the AI Buildout?: MONDAY MAMLMSbraddelong.substack.com

  6. 12 days earlier
  7. SR

    Sebastian Raschka quoted

    AI research engineer working on large language models

    Our readingLooped/recurrent-depth transformer architectures can improve model quality at a fixed compute budget when the model is large enough.

    Their wordsSo, in short, we can say that using looped transformers can improve model quality at a fixed compute budget if the model is large enough.

    ↗GPT-6 Astra, Looped Transformers, and Hidden Reasoningmagazine.sebastianraschka.com 3rd of 3 in this piece

  8. 1 day earlier
  9. SA

    Scott Alexander quoted

    Essayist; writes Astral Codex Ten

    Our readingLarge language models are grown rather than built, and nobody really knows how the resulting working AI functions.

    LLMs

    Their wordsLarge language models are “grown, not built”. Researchers run training data through a neural network. Eventually this creates a working AI; nobody really knows how.

    ↗God Help Us, Let’s Try To Learn About Mechanistic Interpretability Techniquesastralcodexten.com

  11. SA

    Scott Alexander quoted

    Our readingNeural networks do not have a one-to-one mapping between individual neurons and concepts.

    Their wordsresearchers started by seeking a 1:1 mapping between neurons and concepts, like a neuron that always fired when the AI was thinking about cats, but quickly learned that nothing like that existed.

    ↗God Help Us, Let’s Try To Learn About Mechanistic Interpretability Techniquesastralcodexten.com

  12. 1 day earlier
  13. KK

    Kevin Kelly quoted

    Founding executive editor of Wired and author of What…

    Our readingA neural network's latent space is closer to an uncopyrightable syntax than to copyrightable expression.

    LLMs

    Their wordsThe latent spaces of an LLM are closer to something like a syntax which is not copyrightable.

    ↗Arguments in Favor of AI Fair Usekk.org 1st of 3 in this piece

  14. 6 weeks earlier
  15. BC

    Boris Cherny quoted

    Creator of Claude Code at Anthropic

    Our readingPrompt injection can be caught by watching the neurons that fire when it happens, so a defence no longer depends on the model reporting the attack.

    prompt injection

    Their wordsit's it's literally we're looking at neurons in the model's brain that light up when prompt injection happens. So the model won't even tell you but we can actually see those neurons and we can figure out and diagnose that it's happening. And then you combine that with the auto mode classifier and with these three layers we just cannot demonstrate prompt injection anymore.

    ↗Boris Cherny: We Cut 80% of Claude Code's Promptyoutube.com 2nd of 20 in this recording

  16. 5 weeks earlier
  17. LW

    Lilian Weng quoted

    Machine-learning researcher

    Our readingAs models keep growing, AI is running out of enough high-quality unique training tokens to keep up.

    Their wordsAs the model size grows significantly, we are running out of enough high-quality unique tokens.

    ↗Scaling Laws, Carefullylilianweng.github.io

  18. 3 months earlier
  19. AK

    Andrej Karpathy quoted

    Founding member of OpenAI and former director of AI at Tesla

    Our readingAn agent can understand a maximally simplified explanation and still be unable to come up with it — that gap is what is left of the expert's job.

    Their wordsso for example, micro GPT, like I asked I tried to get an agent to write micro GPT. So, I told it like try to boil down the simplest things. Like try to boil down my um neural network training to the simplest thing and it can't do it.

    ↗Skill Issue: Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AIyoutube.com 20th of 20 in this recording

  20. 8 weeks earlier
  21. AW

    Adam Wiggins quoted

    Software developer and entrepreneur; co-founder of Heroku

    Our readingOpen-source text embedding models run well enough on consumer hardware to be practical for real tasks.

    Their wordsLocal / open text embedding models work fine on consumer hardware.

    ↗Email triage with an embedding-based classifieradamwiggins.com 3rd of 3 in this piece

  22. 2 months earlier
  23. AK

    Andrej Karpathy quoted

    Our readingA verifiable task can be optimised by reinforcement learning until a neural network performs it extremely well.

    reinforcement learning

    Their wordsIf a task/job is verifiable, then it is optimizable directly or via reinforcement learning, and a neural net can be trained to work extremely well.

    ↗Verifiabilitykarpathy.bearblog.dev

  24. 4 weeks earlier
  25. AK

    Andrej Karpathy quoted

    Our readingPre-training is a crappy evolution: the practically buildable substitute for the process that gave animals their built-in hardware.

    scaling lawsreinforcement learning

    Their wordsSo that's why I kind of call pre-training this kind of like crappy evolution. It's like the practically possible version with our technology and what we have available to us to get to a starting point where we can actually do things like reinforcement learning and so on.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 5th of 30 in this recording

  27. AK

    Andrej Karpathy quoted

    Our readingThe knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out.

    scaling laws

    Their wordsAnd actually, you don't actually need or want the knowledge. I actually think that's probably actually holding back the neural networks overall because it's actually like getting them to rely on the knowledge a little too much sometimes.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 6th of 30 in this recording

  28. 3 weeks earlier
  29. RS

    Richard Sutton quoted

    Computer scientist who founded the field of reinforcement learning

    Our readingWe have almost no automated techniques for making a system transfer what it learns, and none of the few we have are used in modern deep learning.

    Their wordsWe don't have any methods that are good at that. What we have are people um try different things and they they settle on something that that uh a representation that that transfers well or they generalize as well. But we have no we don't have any automated techniques to promote. we have very few automated techniques to promote transfer and they're not none of them are used in in modern deep learning.

    ↗Richard Sutton – Father of RL thinks LLMs are a dead endyoutube.com 14th of 25 in this recording

  31. RS

    Richard Sutton quoted

    Our readingDeep learning generalises badly, and catastrophic interference with what a network already knew is the proof of it.

    Their wordsso we know deep learning is really bad at this for example we know that if you train on some new thing it will often catastrophically interfere with all the old things that you that you knew

    ↗Richard Sutton – Father of RL thinks LLMs are a dead endyoutube.com 15th of 25 in this recording

  32. 5 weeks earlier
  33. NN

    Neel Nanda quoted

    Runs the mechanistic interpretability team at Google DeepMind

    Our readingComplete reverse-engineering of neural networks is a less promising research direction than studying model biology and applying interpretability usefully

    Their wordsI'm now fairly pessimistic about ambitious interpretability (i.e. complete reverse-engineering), and I'm excited about model biology (studying qualitative high-level properties of models) and applied interpretability (rigorously doing useful things with interp).

    ↗MATS Applications Open (Due Aug 29)neelnanda.io 1st of 3 in this piece

  34. 4 weeks earlier
  35. DH

    Demis Hassabis quoted

    Co-founder and CEO of Google DeepMind

    Our readingAnything nature shaped can be learned efficiently by a classical neural network, because evolutionary processes leave structure behind.

    Their wordsAnd I think the reason that's possible is that in nature, natural systems have structure because they were subject to evolutionary processes that shape them. And if that's true, then you can maybe learn what that structure is.

    ↗Demis Hassabis: Future of AI, Simulating Reality, Physics and Video Games | Lex Fridman Podcast #475youtube.com 1st of 20 in this recording

  36. 3 weeks earlier
  37. NE

    Nelson Elhage quoted

    Software engineer who writes the blog Made of Bugs about performance

    Our readingUsing a neural network is likely to become easier than designing a custom-tuned prediction algorithm for many tasks.

    Their wordsin 2025 it might end up being easier to just throw a neural net at the problem, instead of a custom-designed-and-tuned prediction algorithm!

    ↗The ITTAGE indirect branch predictorblog.nelhage.com 3rd of 3 in this piece

  38. 2 months earlier
  39. LW

    Lilian Weng quoted

    Our readingLonger test-time thinking improves an AI model's robustness to adversarial or unusual inputs.

    Their wordsthinking for longer should be especially useful when the model is presented with an unusual input, such as an adversarial example or jailbreak attempt

    ↗Why We Thinklilianweng.github.io 2nd of 3 in this piece

  40. 3 months earlier
  41. NL

    Nathan Lambert quoted

    Machine-learning researcher on open language models

    Our readingThe Bitter Lesson is misread as being about scale; it is about refusing to add human priors, because a clever specific fix always loses to a simple scalable one.

    Their wordsThe scale word gets a lot of attention in this. The interpretation that I use is effectively to avoid adding the human priors to your learning process. And if you read the original essay, this is what it talks about is how researchers will try to come up with clever solutions to their specific problem that might get them small gains in the short term while simply enabling these deep learning systems to work efficiently, and for these bigger problems in the long term might be more likely to scale and continue to drive success.

    ↗DeepSeek, China, OpenAI, NVIDIA, xAI, TSMC, Stargate, and AI Megaclusters | Lex Fridman Podcast #459youtube.com 2nd of 44 in this recording

  42. 4 months earlier
  43. GM

    Gary Marcus quoted

    Cognitive scientist, professor emeritus of psychology…

    Our readingScaling the 2019-2023 recipe — same architecture, bigger model, more data — is not enough; further progress depends on new architectural ideas.

    Their wordsNeurosymbolic AI — combining such machinery with neural networks – is likely a necessary condition for going forward.

    ↗LLMs don’t do formal reasoning - and that is a HUGE problemgarymarcus.substack.com 2nd of 2 in this piece

  44. 2 months earlier
  45. AG

    Alexey Guzey quoted

    Writer and founder of New Science

    Our readingWork on large language models is not progress toward artificial general intelligence, because the models do not understand what they read.

    measuring intelligenceAGILLMs

    Their wordsThis means that working on neural networks is NOT getting us closer to AGI, except indirectly.

    ↗How my views on AI changed every year 2017-2024guzey.com

  46. 7 weeks earlier
  47. AS

    Aravind Srinivas quoted

    Co-founder and CEO of Perplexity, an AI answer engine

    Our readingThe transformer was close to the last answer in architecture: nothing about the core design has changed since 2017.

    design

    Their wordsAnd that is the, I would say, it’s almost like the last answer. Nothing has changed since 2017 except maybe a few changes on what the nonlinearities are and how the square descaling should be done. Some of that has changed.

    ↗Aravind Srinivas: Perplexity CEO on Future of AI, Search & the Internet | Lex Fridman Podcast #434youtube.com 7th of 20 in this recording

  48. 2 months earlier
  49. SR

    Sebastian Ruder quoted

    Research scientist at Meta working on multilingual models and evaluation

    Our readingThe performance gap between closed-source and open-weight language models is closing.

    LLMs

    Their wordsI'm excited about what this means for the open-source community and research, with the gap between closed-source and open-weight models closing and SOTA-level conversational models being more easily accessible.

    ↗Command R+ruder.io 1st of 2 in this piece

  50. 3 months earlier
  51. CH

    Chip Huyen quoted

    Engineer and writer on machine-learning systems

    Our readingThe probabilistic nature of AI models is what makes them well suited to creative tasks

    Their wordsThis probabilistic nature makes AI great for creative tasks.

    ↗Generation configurations: temperature, top-k, top-p, and test time computehuyenchip.com 1st of 3 in this piece

  52. 3 weeks earlier
  53. AT

    Alex Telford quoted

    Writer on biotechnology and the economics of drug development

    Our readingDeep learning is not yet ready for the task that would actually add value in drug development: predicting safety and efficacy in humans.

    Their wordsit will take time for deep learning systems and training data to mature enough to be useful for the truly value-adding task of predicting safety and efficacy in humans

    ↗The pharma industry from Paul Janssen to today: why drugs got harder to develop and what we can do about itatelfo.github.io 8th of 9 in this piece

  54. 6 months earlier
  55. YB

    Yoshua Bengio quoted

    Deep-learning researcher and Turing Award laureate

    Our readingAccess to large-scale generalist AI systems that could be weaponized should be limited, meaning their code and neural network parameters should not be released open-source.

    Their wordsAt the same time, in order to reduce the probability of someone intentionally or unintentionally bringing about a rogue AI, we need to increase governance and we should consider limiting access to the large-scale generalist AI systems that could be weaponized, which would mean that the code and neural net parameters would not be shared in open-source and some of the important engineering tricks to make them work would not be shared either.

    ↗FAQ on Catastrophic AI Risksyoshuabengio.org

  56. 4 months earlier
  57. AR

    Armin Ronacher quoted

    Creator of Flask and Jinja

    Our readingWe do not control neural networks well enough to guarantee an AI will not harm humans, so we should be careful about what capabilities we give them.

    Their wordsThe biggest issue is probably that we don’t control neutral networks enough to be able to ensure AI doesn’t harm humans. We can’t even control AI to not reveal internal prompts.

    ↗I Think AI Would Kill my Wifelucumr.pocoo.org

  58. 8 months earlier
  59. EY

    Eliezer Yudkowsky quoted

    Co-founder of the Machine Intelligence Research Institute…

    Our readingNobody knows what is going on inside a large model, and the interpretability pictures we can draw do not answer the only question that matters.

    Their wordsWe’ve got no idea what’s actually going on inside the giant inscrutable matrices and tensors of floating-point numbers.

    ↗AGI Ruin: A List of Lethalitiesintelligence.org 7th of 10 in this piece