korrents

korrents · Dwarkesh Podcast

Which AI is the best liar?

Dwarkesh Patel · 18m · youtube.com ↗

8 korrents from this recording

Dwarkesh Patel did not write this page.

Every claim below is a statement made in this recording, quoted word for word and linked to the second it was said, so you can hear it rather than take our word for it. The wording comes from the transcript published alongside the recording; the sentence above each quote is our reading of the claim, not their wording.

  1. 0:11 · watch on youtube.com↗

    watching them play actually lets us learn a lot about how AI is progressing on a set of AI capabilities that no other benchmark that I know of gives you clear visibility on.
  2. 3 min later
  3. 3:15 · watch on youtube.com↗

    What it shows you is that the models, even over the course of the last few months, have gotten much better at long-term coherence, multi-agent coordination, and strategic reasoning.
  4. 4 min later
  5. 7:11 · watch on youtube.com↗

    You'll notice from the leaderboards that OpenAI's newest models are way better at this game than Anthropic's. That wasn't always the case.
  6. 1 min later
  7. 7:53 · watch on youtube.com↗

    The biggest difference is that OpenAI's models are much more skeptical now of any particular piece of evidence [music] and consider it more holistically alongside everything that has happened over the last several days, whereas Anthropic's models tend to take reported information at face [music] value.
  8. 3 min later
  9. 10:31 · watch on youtube.com↗

    It seems like evil's win rate gets slightly lower as both sides get stronger, but that drift is too small to be distinguishable from chance, which is good news for this Blood on the Clocktower evaluation because unlike other benchmarks, this one is unlikely to get saturated.
  10. 1 min later
  11. 11:30 · watch on youtube.com↗

    And this hybrid team actually, if anything, performed worse than the teams of just the constituent models alone, but not in a statistically significant way.
  12. 3 min later
  13. 14:11 · watch on youtube.com↗

    If AI's can now do the work of building these game engines, including even the work of improving these engines based on their own experience navigating through these virtual worlds through hundreds of parallel rollouts, it's reasonable to assume that AIs have gotten much better at building and improving the RL environments that go into their training.
  14. 3 min later
  15. 17:03 · watch on youtube.com↗

    In all the games that I've run so far with current models, no AI has ever used this approach to work out who his teammates were. Will this change with future models? I don't know, but it's worth keeping an eye out.