korrents
Andrej Karpathy

What Andrej Karpathy thinks about scaling laws

@andrej-karpathy · 69 positions · On ppll ↗

Founding member of OpenAI and former director of AI at Tesla; creator of nanoGPT and the term "vibe coding".

Andrej Karpathy did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

Filter scaling laws · Newest first
Topic
All 69LLMs 26reinforcement learning 8AI agents 7AI and jobs 7coding agents 6neural networks 4AGI 3scaling laws 3AI alignment 2
19 more topics
AI and science 2code generation 2computer vision 2education 2AI and human skill 1developer tools 1documentation 1energy 1government 1ideas vs execution 1Microsoft 1open source 1OpenAI 1optimism 1physics 1product management 1self-driving cars 1software quality 1vibe coding 1
Sort
NewestOldest

Only Andrej Karpathy on scaling laws. See everyone on scaling laws

3 dated positions, 2025, in their own words. Our reading of what Andrej Karpathy has said — not written or endorsed by them.

3 positions so far — this page is not yet offered to search engines.

  1. Our readingPre-training is a crappy evolution: the practically buildable substitute for the process that gave animals their built-in hardware.

    reinforcement learning

    Their wordsSo that's why I kind of call pre-training this kind of like crappy evolution. It's like the practically possible version with our technology and what we have available to us to get to a starting point where we can actually do things like reinforcement learning and so on.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 5th of 30 in this recording

  3. Our readingThe knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out.

    neural networks

    Their wordsAnd actually, you don't actually need or want the knowledge. I actually think that's probably actually holding back the neural networks overall because it's actually like getting them to rely on the knowledge a little too much sometimes.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 6th of 30 in this recording

  5. Our readingModels are enormous only because the internet is bad: most of that capacity does memory work, and a better dataset would need far less of it.

    Their wordsUm so I almost feel like because the internet is so terrible, we actually have to sort of like build really big models to compress all that. Uh most of that compression is memory work instead of like cognitive work. But what we really want is the cognitive part actually delete the memory

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 17th of 30 in this recording