korrents
Andrej Karpathy

What Andrej Karpathy thinks about scaling laws

@andrej-karpathy · 66 positions · 0 changes of mind

Founding member of OpenAI and former director of AI at Tesla; creator of nanoGPT and the term "vibe coding".

Everything they publish, on ppll ↗

Andrej Karpathy did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

2 dated positions, 2025, in their own words. Our reading of what Andrej Karpathy has said — not written or endorsed by them.

2 positions so far — this page is not yet offered to search engines.

  1. Our readingPre-training is a crappy evolution: the practically buildable substitute for the process that gave animals their built-in hardware.

    Their wordsSo that's why I kind of call pre-training this kind of like crappy evolution. It's like the practically possible version with our technology and what we have available to us to get to a starting point where we can actually do things like reinforcement learning and so on.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 5th of 30 in this recording

    reinforcement learning

    Interesting

  3. Our readingThe knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out.

    Their wordsAnd actually, you don't actually need or want the knowledge. I actually think that's probably actually holding back the neural networks overall because it's actually like getting them to rely on the knowledge a little too much sometimes.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 6th of 30 in this recording

    Interesting