korrents

Jan Leike

@jan-leike · 12 positions · 0 changes of mind

Alignment researcher. Co-led OpenAI's superalignment team until 2024 and has led alignment work at Anthropic since; writes Musings on the Alignment Problem.

Jan Leike did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

  1. Our reading

    Simple training interventions turned out to be very effective at steering models towards aligned behaviour.

    Their wordsBut the most important lesson is that simple interventions are very effective at steering the model towards more aligned behavior.

    ↗Alignment is not solvedaligned.substack.com 1st of 5 in this piece

    Interesting

  3. Our reading

    Alignment of today's models is going well enough to look solvable, while aligning models we can no longer understand remains unsolved.

    Their wordsThis is the hard problem of alignment we need to solve in order to succeed at building superintelligence, and to this day it is an unsolved problem.

    ↗Alignment is not solvedaligned.substack.com 2nd of 5 in this piece

    AI alignment

    Interesting

  5. Our reading

    We do not have to align superintelligence directly; we have to build a human-level automated alignment researcher we trust more than ourselves.

    Their wordsBut the goal we need to achieve is so much easier: we just need to build a model that’s as good as us at alignment research, and that we trust more than ourselves to do this research well because it’s sufficiently aligned.

    ↗Alignment is not solvedaligned.substack.com 3rd of 5 in this piece

    AI alignment

    Interesting

  7. Our reading

    Recursive self-improvement has already started, because AI research itself is now being automated.

    Their wordsWe are starting to automate AI research and the recursive self-improvement process has begun.

    ↗Alignment is not solvedaligned.substack.com 4th of 5 in this piece

    recursive self-improvement

    Interesting

  9. Our reading

    Making a capable model malicious is about as easy as making it aligned, because the same levers work in both directions.

    Their wordsIn fact, making an evil version of Claude that’s just as smart and agentic would be pretty easy.

    ↗Alignment is not solvedaligned.substack.com 5th of 5 in this piece

    Anthropic

    Interesting
  10. 12 months earlier

  11. Our reading

    We cannot know how much safety a control measure buys until we understand elicitation, so control adds a margin of unknown size.

    Their wordsHowever, without having a good handle on elicitation, we cannot be confident that our control techniques are effective.

    ↗Should we control AI instead of aligning it?aligned.substack.com 1st of 2 in this piece

    Interesting

  13. Our reading

    Building AI that can actually be trusted is the goal; containing an AI known to be misaligned is not a substitute for it.

    Their wordsMore generally, we should actually solve alignment instead of just trying to control misaligned AI.

    ↗Should we control AI instead of aligning it?aligned.substack.com 2nd of 2 in this piece

    AI alignment

    Interesting
  14. 3 months earlier

  15. Our reading

    Releasing open weights is irreversible in a way that deploying a hosted model is not, so its risks cannot be mitigated later.

    Their wordsThis is especially severe for open weight models, where deployment is essentially irreversible and mitigations can’t be improved later.

    ↗Two alignment threat modelsaligned.substack.com 1st of 3 in this piece

    Interesting

  17. Our reading

    Human oversight of AI behaviour is already intractable at the volume models produce, so the overseers have to be models too.

    Their wordsTherefore it’s simply intractable to meaningfully oversee all but a tiny fraction of our models’ behavior with actual humans.

    ↗Two alignment threat modelsaligned.substack.com 2nd of 3 in this piece

    Interesting

  19. Our reading

    Today's models are severely under-elicited: they are far more capable than the way we currently ask them makes them look.

    Their wordsIn other words, current models are probably severely under-elicited.

    ↗Two alignment threat modelsaligned.substack.com 3rd of 3 in this piece

    Interesting
  20. 14 months earlier

  21. Our reading

    Once model weights leave their owner's control the loss is irreversible, which is why securing them is the precondition for governing AI at all.

    Their wordsMoreover, model exfiltration is likely impossible to reverse.

    ↗Self-exfiltration is a key dangerous capabilityaligned.substack.com 1st of 2 in this piece

    Interesting

  23. Our reading

    The dangerous capability that matters most for misalignment is whether a model can exfiltrate its own weights, because that is the one that removes it from your control.

    Their wordsIf a model was capable of self-exfiltration, it would have the option to remove itself from your control.

    ↗Self-exfiltration is a key dangerous capabilityaligned.substack.com 2nd of 2 in this piece

    AI alignment

    Interesting