korrents

korrents · Dwarkesh Podcast

Dylan Patel — The single biggest bottleneck to scaling AI compute

Dylan Patel · 2h 30m · youtube.com ↗

28 korrents from this recording

1h2h
Dylan Patel did not write this page.

Every claim below is a statement made in this recording, quoted word for word and linked to the second it was said, so you can hear it rather than take our word for it. The wording comes from the transcript published alongside the recording; the sentence above each quote is our reading of the claim, not their wording.

  1. 0:15:28 · watch on youtube.com↗

    Our reading

    An H100 is worth more today than it was when it was new, because the models it runs got better faster than its replacements arrived.

    Their wordsif improvement stopped you know here the value of an H100 is now predicated on the value that GPD 5.4 four can get out of it instead of the value that GP4 can get out of it and the margins and all that stuff that these labs are doing and they're in a competitive environment so their margins can't go to infinity. Um so you sort of have this like dynamic that is quite interesting in that an H100 is worth more today than it was 3 years ago.
    Interesting
  2. 6 min later

  3. 0:21:32 · watch on youtube.com↗

    Our reading

    Committing to compute years in advance is the durable margin advantage in AI, because every increment of new capacity transacts at the new price.

    Their wordsA companies that have locked up, you know, and and don't have commitment issues, you know, have these 5-year contracts for compute, they've kind of locked in a humongous margin advantage because they've locked in compute for 5 years at a price of what it transacted at 5 years ago or three years ago or two years ago, whatever it is.
    Interesting
  4. 3 min later

  5. 0:24:36 · watch on youtube.com↗

    Our reading

    The model companies' margins will rise this year, because being capacity-constrained forces them to destroy their own demand.

    Their wordsUm I think at least this year we're going to see margins for the model vendors go up a lot, right? Because they're so capacity constrained, they have to demand destroy demand, right? there is there's no way they can continue anthropic can continue at the current pace without destroying demand.
    Interesting
  6. 3 min later

  7. 0:27:20 · watch on youtube.com↗

    Our reading

    TSMC would rather give wafers to a slow, stable CPU business than to its fastest-growing AI customer.

    Their wordsTSMC is much more excited to give allocation to Graviton than they are to tranium because they view CPU business as more stable long-term growth right and as a company that is conservative and doesn't want to ride cycles of growth too hard you actually want to allocate to the uh the market that is more stable and lower growth rate first before you allocate all the incremental capacity to the fast growth rate market.
    Interesting
  8. 5 min later

  9. 0:32:25 · watch on youtube.com↗

    Our reading

    Anthropic saw the coming shortage of chips before Google did, and bought Google's own TPUs out from under it.

    Their wordsBecause Enthropic saw it before Google. And then Google had Nano Banano and Gemini 3 which caused their user metrics to skyrocket and leadership at Google was like oh and then they started making the statement of we have to double compute every is it 6 months or I don't remember the exact number that they said.
    Interesting
  10. 2 min later

  11. 0:34:51 · watch on youtube.com↗

    Our reading

    The binding constraint on AI is no longer power or data centres but the semiconductor supply chain, which has by far the longest lead times.

    Their wordsYeah, I think the biggest bottleneck is compute and for that the longest lead time supply chains are not power or data centers. They're actually the semiconductor supply chain themselves, right? It switches back from being power and data center uh as a major bottleneck to chips.
    Interesting
  12. 2 min later

  13. 0:37:03 · watch on youtube.com↗

    Our reading

    By the end of this decade the ceiling on AI compute is set by ASML, the lowest rung of the supply chain, and not by anything above it.

    Their wordsSo to scale compute further right there's some different bottlenecks this year next year uh but ultimately by 2829 the bottleneck falls to the lowest rung on the supply chain which is ASML right ASML makes the world's most complicated machine i.e. an EUV tool.
    Interesting
  14. 4 min later

  15. 0:40:48 · watch on youtube.com↗

    Our reading

    Hundreds of billions of dollars of AI value rest on about a billion dollars of lithography tooling that cannot be scaled quickly.

    Their wordsoh 50 gigawatts of economic you know sort of capex in in the data center and what gets built on top of that in terms of tokens is even larger right it might be hundred billion dollars worth of AI value into the supply chain is held up by this $1.2 two billion dollars worth of tooling that simply just cannot expand its supply chain quickly.
    Interesting
  16. 2 min later

  17. 0:42:25 · watch on youtube.com↗

    Our reading

    The world can make roughly 200 gigawatts of AI chips a year by 2030, which makes a gigawatt a week a plausible quarter of the market rather than a fantasy.

    Their wordsUm, and then you stack on 70 this year, 80 next year, growing to 100 by 2030. You're at like 700 EV tools by the end of the decade. Um, 700 EV tools, three and a half tools per gigawatt. um assuming it's all allocated to AI which it's not but three and a half tools per gigawatt gets you to 200 gigawatts worth of AI chips for the data centers to deploy
    Interesting
  18. 3 min later

  19. 0:45:42 · watch on youtube.com↗

    Our reading

    ASML has never raised the price of a tool by more than it raised the tool's capability, which makes it the most generous monopoly in the industry.

    Their wordsYou can you can take the margin like Nvidia takes the margin. memory players are taking the margin, but ASML has never risen the price more than they've increased the capability of the tool. Um, and so in a sense, they've always provided net benefit to their customer.
    Interesting
  20. 1 min later

  21. 0:46:35 · watch on youtube.com↗

    Our reading

    The semiconductor supply chain will underbuild not because it cannot expand but because nobody in it believes the demand is real.

    Their wordsUm in general the semiconductor supply chain has not right it's lived through the booms and bust and uh we can talk a bit more about it but basically no one you know some players as of very recently have like woken up but in general no one really sees demand for 200 gawatts a year of AI chips or you know trillions of dollars of spend a year in the semiconductor supply chain.
    Interesting
  22. 16 min later

  23. 1:02:11 · watch on youtube.com↗

    Our reading

    Falling back to older 7-nanometre chips would not rescue AI compute, because the real gap between chip generations is twentyfold, not the threefold the flops suggest.

    Their wordsSo when you look at inference at let's say 100 tokens a second for deepseek and kimk 2.5 hopper versus blackwell the performance difference is on the order of 20x
    Interesting
  24. 5 min later

  25. 1:07:30 · watch on youtube.com↗

    Our reading

    China will have a working EUV machine long before it can build them in quantity, because production hell takes years after the lab result.

    Their wordsI think they'll have working tools. I don't think that they'll be able to manufacture a bunch yet, right? You know, there's they're sort of having it work and then there's production hell, right?
    Interesting
  26. 4 min later

  27. 1:11:44 · watch on youtube.com↗

    Our reading

    As labs sell finished white-collar work rather than reasoning traces, Chinese models will lose the ability to distil their way to parity.

    Their wordsAs we move from, you know, hey, [clears throat] these companies are selling tokens where they provide the entire uh reasoning chain and all that to uh selling automated, you know, white collar work, right? Automated software engineer, send them the request, they give you the result back and there's a bunch of thinking on the back end that they don't show you. The ability to distill out of American models into Chinese models will be harder.
    Interesting
  28. 4 min later

  29. 1:15:51 · watch on youtube.com↗

    Our reading

    You do not have to believe in AGI to believe in the timelines on which the United States wins the semiconductor race.

    Their wordsBut I don't know like I don't know what fast timelines means, right? Like I I like don't think you have to believe in AGI to have the timelines where the US wins.
    Interesting
  30. 2 min later

  31. 1:17:38 · watch on youtube.com↗

    Our reading

    Nobody wants a slow model, which is why labs will not trade inference speed for cheaper memory even though they easily could.

    Their wordsThey could release claw slow mode and have an increase in tokens per dollar by a significant amount. Um they could probably like reduce the price of Opus 46 by you know 4x 5x and reduce the speed by another by maybe just like 2x like the curve on inference throughput versus speed is there already just on hm um and yet they don't um because no one actually wants to use a slow model
    Interesting
  32. 5 min later

  33. 1:22:09 · watch on youtube.com↗

    Our reading

    What an accelerator needs from memory is bandwidth per unit of chip edge, and on that measure ordinary DRAM is an order of magnitude behind HBM.

    Their wordsand even if you take a generous interpretation of 128 * 8 gig transfers, you're at 128 gigabytes a second for the same shoreline versus 2 and a half terabytes a second. There's a there's an order of magnitude difference in bandwidth per edge area.
    Interesting
  34. 5 min later

  35. 1:26:55 · watch on youtube.com↗

    Our reading

    The memory crunch will make ordinary phones and computers worse year on year, and that is why the public will come to hate AI more.

    Their wordsUm DRAM gets released goes to AI chips who are willing to do longer term contracts, willing to pay higher margins, etc., etc. because at the end of the day, the margin that they extract is much larger from the end user or whatever. Um, and so this this this probably leads to like people hating AI even more, right?
    Interesting
  36. 4 min later

  37. 1:30:38 · watch on youtube.com↗

    Our reading

    The reason we cannot simply make more memory is that there is nowhere to put the tools, and a fab takes two years to build.

    Their wordsUh, Micron bought a fab from a company in Taiwan that makes lagging edge chips, right? Um, Heinix and Samsung are doing, you know, some pretty crazy things to try and expand capacity at their existing fabs, uh, that also have like very large knock-on effects in the economy. And so, hey, why can't we build more capacity is like there's nowhere to put the tools, right?
    Interesting
  38. 5 min later

  39. 1:35:12 · watch on youtube.com↗

    Our reading

    Elon Musk can probably build the cleanroom for a million-wafer fab, but he cannot develop the process technology quickly, because that knowledge is accumulated rather than invented.

    Their wordsbut I think he can build the uh clean room. It'll take a year or two. Maybe initially it won't be super fast, but then over time you'll get faster and faster at it. But then the really complex part is actually developing the process technology and building wafers. And I don't think he can develop that uh quickly. I think that has a lot of built-up knowledge.
    Interesting
  40. 12 min later

  41. 1:46:51 · watch on youtube.com↗

    Our reading

    About a fifth of the US grid could be freed for data centres just by covering the yearly peak with batteries or peaker plants.

    Their wordsThen all of a sudden, you've unlocked 20% of the US grid for data centers because most of the times that capacity is sitting idle and it's really only there for that peak, right? Which is a day or two, right?
    Interesting
  42. 5 min later

  43. 1:52:02 · watch on youtube.com↗

    Our reading

    Labour is a huge constraint on the AI build-out, and the way round it is to build data centres as modules in Asian factories rather than on site.

    Their wordsum humanoid robots maybe start to or robotics at least start to but the main factor is going to be for reducing the number of people is modularizing things and making them in factories in Asia
    Interesting
  44. 5 min later

  45. 1:57:13 · watch on youtube.com↗

    Our reading

    Getting a data centre permitted in the empty middle of America is far easier than putting one in orbit.

    Their wordsAnd so I think people are figuring out how to build these things and permitting like I I just like ultimately like permitting and red tape in middle of nowhere Texas or middle of nowhere Wyoming or middle of nowhere like New Mexico is probably a hell of a lot easier than sending stuff into space
    Interesting
  46. 5 min later

  47. 2:02:13 · watch on youtube.com↗

    Our reading

    Space data centres solve the wrong problem, because the contended resource is chips rather than energy, and that will not change this decade.

    Their wordsSo, space data centers effectively are not li limited by, you know, hey, we have this energy advantage. It's actually just limited by the same contended resource. We can only make 200 gawatts of chips a year by the end of the decade.
    Interesting
  48. 9 min later

  49. 2:10:54 · watch on youtube.com↗

    Our reading

    A lab should spend most of its compute on research rather than on building the next model, because research is where the tenfold yearly efficiency gains come from.

    Their wordswell the model the compute efficiency gains you get from research are so large you actually want most of your compute to go to research not to development because you know all these researchers are generating new ideas trying them out testing them and continuing to march along this and push the prao optimal curve of scaling laws further and further and further
    Interesting
  50. 9 min later

  51. 2:19:46 · watch on youtube.com↗

    Our reading

    TSMC will never throw Apple off its leading node; Apple will simply stop mattering enough for TSMC to cater to it.

    Their wordsand so I don't think TSMC would kick out Apple. I think Apple will become a smaller and smaller and smaller percentage of TSMC's revenue and therefore be less relevant for TSMC to cater to their demands.
    Interesting
  52. 4 min later

  53. 2:23:30 · watch on youtube.com↗

    Our reading

    Huawei, if it had access to TSMC, would arguably be a better chip company than Nvidia.

    Their wordsAnd Huawei has a bigger pool in China. It's very arguable that Huawei, if they had TSMC, would be better than Nvidia.
    Interesting
  54. 6 min later

  55. 2:29:35 · watch on youtube.com↗

    Our reading

    Evacuating TSMC's engineers and destroying its fabs would leave China with the strongest semiconductor supply chain in the world.

    Their wordsUm, just shipping out all the engineers and blowing up the fabs means China has a stronger semiconductor supply chain than the rest of the world, right?
    Interesting