korrents

What Benedict Evans thinks about LLMs

@benedict-evans · 16 positions · 0 changes of mind

Technology analyst whose weekly email on tech reaches over 150,000 readers. Publishes essays and an annual presentation on where the industry is going.

Everything they publish, on ppll ↗

Benedict Evans did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

5 dated positions, 2025, in their own words. Our reading of what Benedict Evans has said — not written or endorsed by them.

  1. Our readingProprietary data is not an advantage in training large language models, because no company's holdings are large enough to matter.

    Their wordsI think it's actually the opposite which is that everyone's kind of using the same data which is you need such an enormous amount of generalized text that the amount that Google has or that meta has is not actually enough to move to be a kind of fundamental difference in what you can train with.

    ↗Why Everyone Is Wrong About AI (Including You) | Benedict Evansyoutube.com 5th of 16 in this recording

    Google

  3. Our readingStudies showing a language model behaving badly prove nothing, because the model was asked to produce that behaviour.

    Their wordsyou write murder is good on a piece of paper and you put it in a photocopier and you press go and you say my god the machine says murder's good. Well, no, you told the machine to say that. And that's what these anthropic studies are. They're basically you tell the machine to say a thing and then it says it like well you haven't proved anything.

    ↗Why Everyone Is Wrong About AI (Including You) | Benedict Evansyoutube.com 6th of 16 in this recording

    Anthropic

  5. Our readingA statistical system that knows when it is wrong is a contradiction in terms.

    Their wordswill the error rate ever be controllable or manageable will you ever get to a model that knows when it's wrong which to me seems like given a stat statistical system seems like a contradiction in terms

    ↗Why Everyone Is Wrong About AI (Including You) | Benedict Evansyoutube.com 10th of 16 in this recording

  7. Our readingLarge language models have no network effect, and remembering a user is a switching cost rather than one.

    Their wordsthere's no apparent equivalent in LLM right now there's no reason why the LMS get better because more people use them now that may come you have that open AI and people have doing memory where it remembers what else you've asked, but that seems more like a switching cost than a network effect.

    ↗Why Everyone Is Wrong About AI (Including You) | Benedict Evansyoutube.com 12th of 16 in this recording

  9. Our readingA language model cannot produce work that is both original and good, because nothing gives it a score for originality.

    Their wordsYou see this for the music now. You can generate new music. It could generate new stuff that you wouldn't know. For an LLM variance is bad. originality is is is a lower score. So what's the feedback loop for original but good?

    ↗Why Everyone Is Wrong About AI (Including You) | Benedict Evansyoutube.com 14th of 16 in this recording