korrents
Andrej Karpathy

What Andrej Karpathy thinks about LLMs

@andrej-karpathy · 69 positions · On ppll ↗

Founding member of OpenAI and former director of AI at Tesla; creator of nanoGPT and the term "vibe coding".

Andrej Karpathy did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

Filter LLMs · Newest first
Topic
All 69LLMs 26reinforcement learning 8AI agents 7AI and jobs 7coding agents 6neural networks 4AGI 3scaling laws 3AI alignment 2
19 more topics
AI and science 2code generation 2computer vision 2education 2AI and human skill 1developer tools 1documentation 1energy 1government 1ideas vs execution 1Microsoft 1open source 1OpenAI 1optimism 1physics 1product management 1self-driving cars 1software quality 1vibe coding 1
Sort
NewestOldest

Only Andrej Karpathy on LLMs. See everyone on LLMs

26 dated positions, 2025 to 2026, in their own words. Our reading of what Andrej Karpathy has said — not written or endorsed by them.

  1. Our readingThe gap in shared understanding of LLM capability is widening.

    Their wordsDusting off this tweet from April because this gap in shared understanding of LLM capability is *widening*.

    ↗@karpathy on Xx.com

  2. 9 days earlier

  3. Our readingLLMs will do more legwork autonomously

    Their wordsAs LLMs get better, they will do more and more of the legwork autonomously, and a lot more of our work will rise up the abstractions into oversight and understanding.

    ↗@karpathy on Xx.com

  4. 6 months earlier

  5. Our readingPeople disagreeing about AI capability are speaking past each other, because they are using models of very different tiers on very different kinds of task.

    Their wordsTLDR the people in these two groups are speaking past each other.

    ↗@karpathy on Xx.com

  6. 12 days earlier

  7. Our readingBecause an LLM can argue almost any direction competently, the right way to use one for forming an opinion is to make it argue every side.

    Their wordsThe LLMs may elicit an opinion when asked but are extremely competent in arguing almost any direction. This is actually super useful as a tool for forming your own opinions, just make sure to ask different directions and be careful with the sycophancy.

    ↗@karpathy on Xx.com

  8. 3 days earlier

  9. Our readingLLM memory as currently built is a distraction to the model: one old question keeps resurfacing as if it were a lasting interest.

    Their wordsOne common issue with personalization in all LLMs is how distracting memory seems to be for the models. A single question from 2 months ago about some topic can keep coming up as some kind of a deep interest of mine with undue mentions in perpetuity.

    ↗@karpathy on Xx.com

  10. 5 days earlier

  11. Our readingThe engineer's resource constraint has moved from flops to tokens: what matters now is the token throughput you command.

    Their wordsBut now it's not about flops, it's about tokens. So what is your token throughput and what token throughput do you command?

    ↗Skill Issue: Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AIyoutube.com 3rd of 20 in this recording

  13. Our readingA swarm of untrusted volunteers on the internet could improve models and run circles around the frontier labs, because the earth has far more compute than they do.

    Their wordsa swarm of agents on the internet could collaborate to improve LLMs and could potentially even like run circles around frontier labs. Like who knows, you know? Um yeah, like maybe that's even possible. Like frontier labs have a huge amount of trusted compute but the earth is much bigger and has huge amount of untrusted compute.

    ↗Skill Issue: Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AIyoutube.com 13th of 20 in this recording

  14. 5 weeks earlier

  15. Our readingA tiny language model inventing a plausible-sounding name is the same phenomenon as a large one confidently stating a false fact.

    OpenAI

    Their wordsmicrogpt “hallucinating” a name like “karia” is the same phenomenon as ChatGPT confidently stating a false fact.

    ↗microgptkarpathy.github.io

  16. 8 weeks earlier

  17. Our readingLLM intelligence should not be understood through an animal analogy; it is a different kind of entity altogether

    Their words2025 is where I (and I think the rest of the industry also) first started to internalize the "shape" of LLM intelligence in a more intuitive sense. We're not "evolving/growing animals", we are "summoning ghosts".

    ↗2025 LLM Year in Reviewkarpathy.bearblog.dev

  18. 2 months earlier

  19. Our readingAgents will take a decade rather than a year, because today's models are cognitively lacking in too many independent ways at once.

    Their wordsSo like they don't have enough intelligence. They're not multimodal enough. They can't do computer use and all this kind of stuff. And they don't do a lot of things. You know, they don't have continual learning. You can't just tell them something and they'll remember it. And they're just cognitively lacking. And it's just not working. And I just think that it will take about a decade to work through all of those issues.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 1st of 30 in this recording

  21. Our readingComputer-use agents had to wait for language models: without pre-trained representations the reward is too sparse to ever learn from.

    Their wordsAnd it just so turns out that um this was extremely early, way too early. so early that we shouldn't have been working on that, you know, uh because um if you're just stumbling your way around and keyboard mashing and mouse clicking and trying to get rewards in these environments, um your reward is too sparse and you just won't learn and you're going to burn a forest uh computing and you're never actually going to get something off the ground.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 2nd of 30 in this recording

  23. Our readingWe are not building animals, we are summoning ghosts: entities made by imitating human text rather than by evolution.

    Their wordswe're not doing training by evolution. Uh we're doing training by basically imitation of humans and the data that they've put on the internet. And so you end up with these like sort of ethereal spirit entities because they're fully digital and they're kind of like mimicking humans.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 3rd of 30 in this recording

  25. Our readingPre-training is a crappy evolution: the practically buildable substitute for the process that gave animals their built-in hardware.

    scaling lawsreinforcement learning

    Their wordsSo that's why I kind of call pre-training this kind of like crappy evolution. It's like the practically possible version with our technology and what we have available to us to get to a starting point where we can actually do things like reinforcement learning and so on.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 5th of 30 in this recording

  27. Our readingThe knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out.

    scaling lawsneural networks

    Their wordsAnd actually, you don't actually need or want the knowledge. I actually think that's probably actually holding back the neural networks overall because it's actually like getting them to rely on the knowledge a little too much sometimes.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 6th of 30 in this recording

  29. Our readingWhat is in a model's weights is a hazy recollection of things it read a year ago; what is in its context window is working memory.

    Their wordsanything that's in the weights it's kind of like a hazy recollection of what you read a year ago anything that you give it as a context uh at test time is directly in the working memory

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 7th of 30 in this recording

  31. Our readingLanguage models have no equivalent of sleep: nothing takes the day's context, thinks it over, and distils it back into the weights.

    Their wordsThese models don't really have this distillation phase um of taking what happened, analyzing it, obsessively thinking through it, um basically doing some kind of a synthetic data generation process and distilling it back back into the weights

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 8th of 30 in this recording

  33. Our readingCoding models are worst at exactly the thing an AI research explosion would need: code that has never been written before.

    Their wordsI think um yeah, they're not very good at code that hasn't never been written before maybe is like one way to put it, which is like what we're trying to achieve when we're building these models.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 10th of 30 in this recording

  35. Our readingProcess supervision by an LLM judge cannot be run for long, because a judge with billions of parameters is gameable and RL will find its cracks.

    reinforcement learning

    Their wordsthe reason that I think this is kind of tricky is quite subtle. And it's the fact that anytime you use an LLM to assign a reward, those LLMs are giant things with billions of parameters and they're gameable.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 14th of 30 in this recording

  37. Our readingEvery sample a model gives you is silently collapsed: each one looks fine, but together they occupy a tiny sliver of the possible space.

    Their wordsall of the samples you get from models are silently collapsed. They're silently, this is not obvious if you look at any individual example of it. They occupy a very tiny manifold of the possible space of um sort of thoughts about content.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 15th of 30 in this recording

  39. Our readingHumans collapse over the course of a life exactly as models do; children say shocking things because they have not overfitted yet.

    Their wordsI also think humans collapse over time. Uh I think this is uh again these analogies are surprisingly good but humans collapse during the course of their lives. This is why children have completely u you know they haven't overfit yet and they will say stuff that will shock you

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 16th of 30 in this recording

  41. Our readingModels are enormous only because the internet is bad: most of that capacity does memory work, and a better dataset would need far less of it.

    Their wordsUm so I almost feel like because the internet is so terrible, we actually have to sort of like build really big models to compress all that. Uh most of that compression is memory work instead of like cognitive work. But what we really want is the cognitive part actually delete the memory

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 17th of 30 in this recording

  43. Our readingCoding is the first job AI took because coding has always been text, which is the one thing these models are built to process.

    Their wordsso I think there's there's an interesting point here because I do believe coding is like the perfect first thing for uh for a for uh these LLMs and uh agents and that's because coding has always fundamentally uh worked around text.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 21st of 30 in this recording

  45. Our readingLanguage models have no culture — no written record they hand to one another — and that absence is one of the things holding them back.

    Their wordsBut I guess like the notion of culture and of written record and of like passing down notes between each other. I don't think there's an equivalent of that with LM right now. So LM don't really have culture right now and it's kind of like one of the I think uh impediments I would say.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 24th of 30 in this recording

  47. Our readingMost of the value an AI expert brings to a company is telling it not to use AI.

    Their wordsa lot of my times the value that I brought to the company was telling them not to use AI.

    ↗Andrej Karpathy — “We’re summoning ghosts, not building animals”youtube.com 27th of 30 in this recording

  48. 6 months earlier

  49. Our readingLLMs reverse the usual pattern of technology diffusion: they benefit ordinary individuals far more than they benefit corporations and governments.

    Their wordsSo it strikes me as quite unique and remarkable that LLMs display a dramatic reversal of this pattern - they generate disproportionate benefit for regular people, while their impact is a lot more muted and lagging in corporations and governments.

    ↗Power to the people: How LLMs flip the script on technology diffusionkarpathy.bearblog.dev

  51. Our readingBecause an individual is an expert in at most one thing, an LLM's broad shallow expertise lets them do things they could not do before, whereas an organisation only gets better at what it already did.

    Their wordsIn contrast, an individual will usually only be an expert in at most one thing, so the broad quasi-expertise offered by the LLM fundamentally allows them to do things they couldn't do before.

    ↗Power to the people: How LLMs flip the script on technology diffusionkarpathy.bearblog.dev