korrents

korrents · piece

Two alignment threat models

Jan Leike · 8 Nov 2024

Read the piece on aligned.substack.com ↗

3 korrents from this piece ↓

Jan Leike did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

  1. Our reading

    Releasing open weights is irreversible in a way that deploying a hosted model is not, so its risks cannot be mitigated later.

    Mental modelOne-way and two-way doors

    Their wordsThis is especially severe for open weight models, where deployment is essentially irreversible and mitigations can’t be improved later.

  2. Our reading

    Human oversight of AI behaviour is already intractable at the volume models produce, so the overseers have to be models too.

    Their wordsTherefore it’s simply intractable to meaningfully oversee all but a tiny fraction of our models’ behavior with actual humans.

  3. Our reading

    Today's models are severely under-elicited: they are far more capable than the way we currently ask them makes them look.

    Their wordsIn other words, current models are probably severely under-elicited.

Keep reading

The other pieces are the ones whose claims are worded most like this one's. That measures language, not agreement: they may argue the opposite.