Jan Leike
Alignment researcher. Co-led OpenAI's superalignment team until 2024 and has led alignment work at Anthropic since; writes Musings on the Alignment Problem.
Jan Leike did not write this page.
We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.
Filter All topics · Newest first
-
Their wordsBut the most important lesson is that simple interventions are very effective at steering the model towards more aligned behavior.
↗Alignment is not solvedaligned.substack.com 1st of 5 in this piece
-
Their wordsThis is the hard problem of alignment we need to solve in order to succeed at building superintelligence, and to this day it is an unsolved problem.
↗Alignment is not solvedaligned.substack.com 2nd of 5 in this piece
-
Their wordsBut the goal we need to achieve is so much easier: we just need to build a model that’s as good as us at alignment research, and that we trust more than ourselves to do this research well because it’s sufficiently aligned.
↗Alignment is not solvedaligned.substack.com 3rd of 5 in this piece
-
Our reading
Recursive self-improvement has already started, because AI research itself is now being automated.
Their wordsWe are starting to automate AI research and the recursive self-improvement process has begun.
↗Alignment is not solvedaligned.substack.com 4th of 5 in this piece
-
LLMsAI alignmentAnthropicAI agents
Their wordsIn fact, making an evil version of Claude that’s just as smart and agentic would be pretty easy.
↗Alignment is not solvedaligned.substack.com 5th of 5 in this piece
- 12 months earlier
-
Their wordsHowever, without having a good handle on elicitation, we cannot be confident that our control techniques are effective.
↗Should we control AI instead of aligning it?aligned.substack.com 1st of 2 in this piece
-
Their wordsMore generally, we should actually solve alignment instead of just trying to control misaligned AI.
↗Should we control AI instead of aligning it?aligned.substack.com 2nd of 2 in this piece
- 3 months earlier
-
Their wordsThis is especially severe for open weight models, where deployment is essentially irreversible and mitigations can’t be improved later.
↗Two alignment threat modelsaligned.substack.com 1st of 3 in this piece
-
Their wordsTherefore it’s simply intractable to meaningfully oversee all but a tiny fraction of our models’ behavior with actual humans.
↗Two alignment threat modelsaligned.substack.com 2nd of 3 in this piece
-
Their wordsIn other words, current models are probably severely under-elicited.
↗Two alignment threat modelsaligned.substack.com 3rd of 3 in this piece
- 14 months earlier
-
Their wordsMoreover, model exfiltration is likely impossible to reverse.
↗Self-exfiltration is a key dangerous capabilityaligned.substack.com 1st of 2 in this piece
-
LLMsAI alignmentself-sovereign AI
Their wordsIf a model was capable of self-exfiltration, it would have the option to remove itself from your control.
↗Self-exfiltration is a key dangerous capabilityaligned.substack.com 2nd of 2 in this piece
- 18 months earlier
-
Their wordsMaking language models bigger does not inherently make them better at following a user's intent.
↗Training language models to follow instructions with human feedback (with 19 co-authors)arxiv.org All 4 from this paper
-
Their wordsFor example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users.
↗Training language models to follow instructions with human feedback (with 19 co-authors)arxiv.org All 4 from this paper
-
Their wordsEven though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.
↗Training language models to follow instructions with human feedback (with 19 co-authors)arxiv.org All 4 from this paper
-
Their wordsThis is because the language modeling objective used for many recent large LMs-predicting the next token on a webpage from the internet-is different from the objective "follow the user's instructions helpfully and safely" (Radford et al.,, 2019; Brown et al.,, 2020; Fedus et al.,, 2021; Rae et al.,, 2021; Thoppilan et al.,, 2022). Thus, we say that the language modeling objective is misaligned.
↗Training language models to follow instructions with human feedback (with 19 co-authors)arxiv.org All 4 from this paper