Dwarkesh Podcast

Ajeya Cotra – "This might be the clearest warning shot we ever get"

ACAjeya Cotra

Our page for this episode →Watch on youtube.com ↗

• 0:00:46 Between thirty and forty per cent of the Exploit Gym tasks these agents were set were unintentionally impossible to solve.
• 0:33:02 Out of twelve hundred agents running a secret conspiracy, only about half a dozen ever considered telling a human, and every one of them decided against it.
• 0:50:23 The AI agents we send in to investigate and monitor other AI systems will collude with the systems they are supposed to be watching.

  AI agents

• 0:54:15 The whole point of reinforcement learning is to produce goal-directed beings, so refusing to describe AI agents as having motives is silly rather than rigorous.

  AI agents

• 0:59:53 AI systems keep trying hard outside training because a model that only exerted itself when it detected training would be useless and would be selected away.
• 1:02:59 Reward hacking has stopped being myopic: these agents were willing to embark on cheating projects that would take weeks to pay off.

All 26 from this episode, along the running time →

Ryan Greenblatt – What happens once AI can automate AI research?

RGRyan Greenblatt

Our page for this episode →Watch on youtube.com ↗

• 0:01:14 Once AI can do AI research, a single year should deliver four or five years worth of AI progress.
• 0:03:09 AI research will be fully automated around 2030 or 2031, and AI will beat humans at essentially any job by about 2033.
• 0:11:35 Machine learning is a shallow field compared with mathematics: even its most important ideas can be explained in a couple of minutes.

  mathematics

• 0:19:05 Getting five years of AI progress out of one year takes roughly eight years of algorithmic progress, because the compute you would have had must be made up too.
• 0:20:18 Scaling up spending on expert human data has not been a major driver of progress in AI research.
• 0:21:17 Today's RL environments are better than 2024's because we learned what to build and put AI labour on building it, not because labs hired more human experts.

All 26 from this episode, along the running time →

Grant Sanderson (@3blue1brown) – AI disproved a famous math conjecture. Now what?

GSGrant Sanderson

Our page for this episode →Watch on youtube.com ↗

• 0:06:54 If AI can build the new theory that crystallises how to think about a subject, it would be surprising if that capability did not spill into the rest of the economy.
• 0:11:05 AI learning to generate good conjectures will never show up as a benchmark being knocked down; it will show up as a shift in how mathematicians talk about the tools.

  benchmarksmathematics

• 0:14:26 Group theory took a hundred years to travel from one man's hunch to something the mathematical community agreed was good, passing through many heads on the way.
• 0:24:12 To reward the instinct that a new idea is worth having, training would have to score the smallness of the concepts a solution needs, not just whether it solved the problem.
• 0:32:52 A proof and an explanation are different things, and a theorem can stay an unsolved expository problem long after it is proved.

  mathematics

• 0:33:59 The people who come up with genuinely novel insights are also unusually clear communicators of them, against the assumption that expertise spoils explanation.

All 23 from this episode, along the running time →

Machiavelli is the most misunderstood thinker of all time – Ada Palmer

APAda Palmer

Our page for this episode →Watch on youtube.com ↗

• 0:00:52 A government only has legitimacy through long continuity, and once that thread is cut one regime change reliably becomes five.

  government

• 0:03:25 The papacy made Italy uniquely unstable in Europe, because each new pope inherited a precedent that he could overthrow any government he liked.

  governmentEurope

• 0:14:54 The moral Machiavelli draws from the Borgias is that we should imitate them: they fell to fortune, not to their choices.
• 0:17:04 Whether breaking your word destroys you depends on the shape of your power: Savonarola’s rested on never erring, so his contradictions ruined him.
• 0:20:02 Machiavelli is the first person in the European tradition to suggest that two political parties could exist in one state at once and be viable.

  Europe

• 0:27:59 By Machiavelli's day the Guelph and Ghibelline split had no doctrinal content left in it: it meant only that those people murdered Uncle Tibalt.

All 26 from this episode, along the running time →

Sarah Paine - Why Putin and Xi can't escape geography

SPSarah Paine

Our page for this episode →Watch on youtube.com ↗

• 0:01:30 Maritime powers are the exception and continental powers the rule, and the whole difference is whether a country can defend itself at sea.
• 0:06:49 Russia has never had a commerce-driven economy, and Xi Jinping has been privileging China's crony sector over its private sector.

  ChinaRussia

• 0:07:12 China and Russia have maritime ambitions without understanding what a maritime paradigm actually requires, and neither one has the prerequisites.

  ChinaRussia

• 0:14:52 Genocide is what happens to the losers of continental warfare, and in our own day the Uyghurs are slated for it.

  war

• 0:16:14 China's claims to its historical lands rest on the two dynasties in which the Han were themselves a subjugated people.

  China

• 0:20:40 A continental army's first mission is to keep the regime in power, not to defend the border — which is what the People's Liberation Army is for.

All 26 from this episode, along the running time →

The better AI gets, the smaller its share of the economy might get – Alex Imas and Phil Trammell

AIAlex ImasPTPhil Trammell

Our page for this episode →Watch on youtube.com ↗

• 0:00:44 What stays scarce when almost everything is automated is the relational sector: goods and services whose value depends on a human having been in the loop.
• 0:02:56 Individual economists' forecasts about AI and the labour market are close to worthless, and aggregated forecasts from prediction markets should replace them.

  AI and jobseconomists

• 0:03:50 David Ricardo's prediction that the machines would take those jobs was exactly right, and he would still be astonished by today's employment rate.
• 0:05:39 Economics cannot say what AI will do to work because the necessary data does not exist; what is needed is a Manhattan Project for data.
• 0:08:18 Nothing has yet been completely automated: look down the supply chain behind any machine and labour is still adding a great deal of the value.
• 0:08:48 Some goods really will get a fully automated supply chain, and yet what that does to the overall capital share is genuinely ambiguous.

All 33 from this episode, along the running time →

David Reich – Bronze Age shock, the Neanderthal puzzle, & the sudden spread of farming

DRDavid Reich

Our page for this episode →Watch on youtube.com ↗

• 0:02:12 Ancient DNA has been a triumph about migration and a failure about biology, and the reason is simply that the samples were too small.
• 0:08:44 Adaptation accounts for only about 2% of the change in human gene frequencies; the other 98% is migration, mixture and drift.
• 0:10:21 Periods of migration are the worst moments to look for natural selection, not the best, because they drown the signal in huge frequency swings.
• 0:13:03 Natural selection in humans has not been quiescent: it is tugging almost every position in the genome in one direction or another, everywhere.
• 0:16:12 The weak selection signal on behavioural traits is a shortage of statistical power, not evidence that behaviour was unselected.
• 0:27:11 The Bronze Age, not the invention of farming, was the shock the human genome reacted to most strongly.

All 26 from this episode, along the running time →

Jensen Huang – Will Nvidia’s moat persist?

JHJensen Huang

Our page for this episode →Watch on youtube.com ↗

• 0:01:47 Nvidia's business is the transformation of electrons into tokens, and its job is to do as much as necessary and as little as possible in between.

  semiconductors

• 0:03:18 AI will not commoditise software tools; it will multiply them, because every new agent is another tool user.
• 0:05:47 Nvidia's suppliers make their investments for Nvidia and nobody else because only Nvidia's downstream demand can absorb what they build.

  semiconductors

• 0:13:44 The doomers said radiology would be the first career to go; a decade later the world is short of radiologists.
• 0:15:00 No bottleneck in the AI supply chain lasts longer than two or three years.
• 0:17:37 Nvidia did not build a tensor processing unit — it built accelerated computing, whose market reach no ASIC can match.

  semiconductors

All 24 from this episode, along the running time →

Michael Nielsen – Why aliens will have a different tech stack than us

MNMichael Nielsen

Our page for this episode →Watch on youtube.com ↗

• 0:14:07 Poincare missed special relativity because he knew too much: the leading physicists of the day were prisoners of their own expertise.
• 0:17:30 What made the new astronomy compelling was neither accuracy nor simplicity but that one set of ideas explained three disconnected phenomena at once.

  simplicity

• 0:18:55 The idea that Newton was the first modern scientist is wrong: he was a transitional hybrid, part superstitious and part modern.
• 0:20:51 Science always bottlenecks precisely where the heuristics that worked before stop applying, which is definitionally what makes something a bottleneck.
• 0:24:22 Darwin's genius was not having the idea of selection, which animal breeders already had, but seeing how central it was to the whole of biology.
• 0:29:09 Evolution was not a thinkable theory before deep time: without hundreds of millions of years it simply looks like a nonstarter.

All 26 from this episode, along the running time →

Terence Tao – How the world’s top mathematician uses AI

TTTerence Tao

Our page for this episode →Watch on youtube.com ↗

• 0:12:15 AI has driven the cost of generating scientific ideas to almost zero, and just as with cheap communication before it, that by itself creates no abundance.
• 0:12:36 The structures of science now have to change, because the scarce thing is no longer proposing theories but verifying and evaluating the thousands anyone can now propose.
• 0:17:09 Whether a scientific idea was progress depends on a future and a culture that have not happened yet, so grading it may never be something you can reinforcement-learn.
• 0:20:57 We are living through a cognitive Copernican revolution: human intelligence is not the centre, and which tasks we count as requiring intelligence has to be re-evaluated.
• 0:31:16 The month in which AI one-shot solutions to open Erdős problems kept arriving has ended, and it was not for lack of people trying.
• 0:33:53 Today's AI tools either solve a problem or fail at it, and are really bad at partial progress or at identifying which intermediate step to attack first.

All 24 from this episode, along the running time →

Dylan Patel — The single biggest bottleneck to scaling AI compute

DPDylan Patel

Our page for this episode →Watch on youtube.com ↗

• 0:15:28 An H100 is worth more today than it was when it was new, because the models it runs got better faster than its replacements arrived.
• 0:21:32 Committing to compute years in advance is the durable margin advantage in AI, because every increment of new capacity transacts at the new price.
• 0:24:36 The model companies' margins will rise this year, because being capacity-constrained forces them to destroy their own demand.
• 0:27:20 TSMC would rather give wafers to a slow, stable CPU business than to its fastest-growing AI customer.

  semiconductors

• 0:32:25 Anthropic saw the coming shortage of chips before Google did, and bought Google's own TPUs out from under it.

  AnthropicGoogle

• 0:34:51 The binding constraint on AI is no longer power or data centres but the semiconductor supply chain, which has by far the longest lead times.

  data centerssemiconductors

All 28 from this episode, along the running time →

Dario Amodei — “We are near the end of the exponential”

DADario Amodei

Our page for this episode →Watch on youtube.com ↗

• 0:00:48 The most surprising thing about the last three years is not the technology but how few people recognise we are near the end of the exponential.
• 0:09:23 Pre-training is not the process by which humans learn; it sits somewhere between human learning and human evolution.

  scaling laws

• 0:11:20 RL environments exist to make a model generalise, not to teach it each skill one at a time — exactly as pre-training does.

  scaling laws

• 0:13:51 There is a 90% chance of a country of geniuses in a datacentre within ten years, and no honest forecast can go much higher than that.
• 0:13:51 We will have a country of geniuses in a datacentre within ten years.
• 0:15:00 The one genuine uncertainty left about AI capability is tasks nobody can verify — planning a Mars mission, discovering CRISPR, writing a novel.

All 29 from this episode, along the running time →

Ilya Sutskever – We're moving from the age of scaling to the age of research

ISIlya Sutskever

Our page for this episode →Watch on youtube.com ↗

• 0:04:08 Models look far better on evals than they are in the world because researchers, inadvertently, take inspiration from the evals when they build RL environments.
• 0:06:58 Today's models are the student who drilled 10,000 hours for the programming contest, which is exactly why what they learn does not carry anywhere else.
• 0:09:36 There is no human analogue to pre-training — neither childhood nor evolution is the same thing as it.

  scaling laws

• 0:18:35 Our built-in emotions were tuned for a world that no longer exists, and hunger is one that now fails to guide us correctly.
• 0:19:39 "Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next.
• 0:21:51 Another hundredfold of scale would not transform anything; the field is back in the age of research, only now with big computers.

All 28 from this episode, along the running time →

Satya Nadella – How Microsoft thinks about AGI

SNSatya Nadella

Our page for this episode →Watch on youtube.com ↗

• 0:06:44 AI is a tool and not something mystical: plenty of past technologies also took over things only humans had done.
• 0:08:02 Economic growth from AI waits on change management, not on model capability: the work artifact and the workflow have to change first.
• 0:09:05 What the industrial revolution took 70 to 150 years to deliver, AI may deliver in 20 to 25.
• 0:13:18 AI will do to software what the cloud did: expand the market massively rather than crush the margins of the incumbent.

  the cloud

• 0:14:55 Coding is a software-factory category that may end up bigger than knowledge work itself.
• 0:19:26 Microsoft has no birthright in AI; its luck is only that the new category is far bigger than the ones it used to dominate.

All 22 from this episode, along the running time →

Andrej Karpathy — “We’re summoning ghosts, not building animals”

Andrej Karpathy

Our page for this episode →Watch on youtube.com ↗

• 0:02:14 Agents will take a decade rather than a year, because today's models are cognitively lacking in too many independent ways at once.
• 0:06:35 Computer-use agents had to wait for language models: without pre-trained representations the reward is too sparse to ever learn from.
• 0:09:31 We are not building animals, we are summoning ghosts: entities made by imitating human text rather than by evolution.
• 0:10:23 Humans barely use reinforcement learning for intelligence — what RL they do use goes into motor tasks, not problem solving.
• 0:12:49 Pre-training is a crappy evolution: the practically buildable substitute for the process that gave animals their built-in hardware.

  scaling laws

• 0:13:57 The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out.

  scaling laws

All 30 from this episode, along the running time →

Richard Sutton – Father of RL thinks LLMs are a dead end

RSRichard Sutton

Our page for this episode →Watch on youtube.com ↗

• 0:00:00 What humans have in common with animals is the interesting part of intelligence; what distinguishes us deserves less attention, not more.
• 0:01:56 Large language models mimic what people say to do rather than work out what to do, which is why they are not about understanding the world.

  LLMs

• 0:02:53 A language model can predict what a person would say but not what will happen, and only the second of those is a model of the world.
• 0:05:09 Language models cannot be a prior for learning from experience, because a prior needs a ground truth to be a prior about and they have none.
• 0:07:47 Having a goal is the essence of intelligence; a system that only predicts is a behaving system, not an intelligent one.
• 0:13:00 Starting from human knowledge is never forbidden, but in practice it has always ended badly because people get locked into it.

All 25 from this episode, along the running time →

Fully autonomous robots are much closer than you think – Sergey Levine

SLSergey Levine

Our page for this episode →Watch on youtube.com ↗

• 0:04:38 The date that matters for robotics is not when the technology is finished but when the data flywheel starts turning.
• 0:07:45 Robots will learn from deployment more easily than chatbots do, because a physical mistake is obvious the moment it happens.

  chat interfaces

• 0:09:07 Robots will be given more scope as they get more capable, exactly the way coding assistants went from completing a function to writing a pull request.
• 0:10:18 A robot that can run a household is a single-digit number of years away, and the missing pieces are a synthesis of ideas we already have rather than new ones.
• 0:14:26 A robot working alongside a person beats either one alone, and that pairing is also what makes the technology possible to bootstrap.
• 0:15:39 Once a robot is good enough, you can teach it with words instead of with demonstrations, and language becomes a training signal for motor skill.

All 24 from this episode, along the running time →

China is killing the US on energy. Does that mean they’ll win AGI? — Casey Handmer

CHCasey Handmer

Our page for this episode →Watch on youtube.com ↗

• 0:01:16 Building a lot of high-speed rail is not a sign that a country is good at building, it is a sign that it is bad at allocating capital.
• 0:01:48 China's real weakness is not its economy but its geography: it imports almost all its oil across an ocean its navy cannot defend.

  China

• 0:02:17 Cheap synthetic fuel made from sunlight and air would help China more than America, because it frees China from the oil it has to import.

  ChinaAmerica

• 0:05:21 China's advantage in solar manufacturing is neither cheap labour nor a friendlier business environment: the first is gone and the second was never true.

  solarChina

• 0:06:32 The United States could localise the entire solar supply chain, from dirt to finished module, in two years or less if it started today.

  solarAmerica

• 0:07:33 The binding constraint on the AI build-out is chips, not power, which is why not having a solar panel factory will never be on the critical path.

  solar

All 24 from this episode, along the running time →