What Neel Nanda thinks about AI alignment
Runs the mechanistic interpretability team at Google DeepMind; worked on interpretability at Anthropic before that.
Neel Nanda did not write this page.
We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.
Filter AI alignment · Newest first
Only Neel Nanda on AI alignment. See everyone on AI alignment
21 dated positions, 2025, in their own words. Our reading of what Neel Nanda has said — not written or endorsed by them.
-
Their wordsit seems likely that a bunch of alignment relevant properties will be introduced during RL training. Um, not we should not assume this will be true for all such properties but like it seems plausible especially for the things like goal seeking, reward hacking, things like that.
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 3rd of 27 in this recording
-
Their wordsEven if the model later learns to be secretive about it, it seems harder to learn while simultaneously being secretive about it without any prior signal in previous transcripts.
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 4th of 27 in this recording
-
Their wordsAnd yeah, it's great that they checked and reported this prominently because, you know, a really easy way to make a model incredibly safe is to replace it with a rock that always says, "As a large language model, I cannot help you with this."
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 5th of 27 in this recording
-
Their wordsjust read the chain of thought is in my opinion a strong contender for one of our best current safety techniques. I also think it will at some point break and we should be paying really close attention to whether it is breaking
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 7th of 27 in this recording
-
Their wordsSo, in my opinion, the current safety cases people give models basically boil down to um this model seems too dumb to be super dangerous.
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 9th of 27 in this recording
-
Their wordsUm the scenarios are often like pretty artificial, kind of deranged in the same way that like A human trying hard to get the model to act badly can write pretty wild and messy and unrealistic scenarios
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 10th of 27 in this recording
-
Their wordsbecause if it's being trained to ignore prompt injections, one of the ways you could achieve this is by being a more deceptive model in general. And it would be good if we don't train our models in a way that incentivizes deception.
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 12th of 27 in this recording
-
Their wordslike we don't want to reinforce deception in models. That seems bad. That seems the kind of thing that could cause misaligned behavior in general. And thus, if we are training them to do role playing, this plausibly causes badness.
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 15th of 27 in this recording
-
Their wordsI am a big fan of not training the models based on what its thoughts look like.
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 16th of 27 in this recording
-
Their wordsWe obviously need to seriously consider this hypothesis, but like h seems pretty unlikely that models develop the ability to do this well and to not verbalize it without sometimes screwing up.
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 18th of 27 in this recording
-
Their wordsUm, but I think evaluation awareness is a much bigger deal for like alignment audits because we're trying to measure will it do something, not can it do something.
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 19th of 27 in this recording
-
Their wordsthey removed data and environments rather than trying to like add in more to like monkey patch it out. Uh this seems like the right approach to me. I think that if you try to monkey patch something out, this might work, but this also might suppress it or like only suppress a particular facet of it.
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 20th of 27 in this recording
-
Their wordsthis is like actually quite an important baseline in my opinion because if you steer, especially with a high coefficient, it's easy to just make the model kind of deranged.
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 21st of 27 in this recording
-
Their wordsI don't really feel that afraid of Opus causing substantial damage, but I think the main loadbearing belief there is that probably it can't like reliably hide things from its chain of thoughts.
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 25th of 27 in this recording
-
Their wordsAnd it seems like we should expect to see like Yeah, we should expect to see like shittier versions before we see like the scary versions.
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 26th of 27 in this recording
-
Our readingHow aligned current models are depends a lot on things like their degree of reward hacking.
Their wordsI kind of feel like a lot of what affects how aligned I think current models are are like things like their degree of reward hacking
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 27th of 27 in this recording
- 7 days earlier
-
Their wordsI want a bunch of decorrelated techniques that can all provide a consistent picture because I just don't really see any one technique getting to the point of reliability where it alone could justify something dramatic like this model is not safe enough to use.
↗Bitter Lesson-Pilled Interp: A Live Paper Review (Activation Oracles & PCD)youtube.com 11th of 21 in this recording
-
Our readingUsing a model as its own activation oracle introduces correlated risk if the model is misaligned.
Their wordsI think another thing that I'm a little bit sketched out by is using the model itself as the activation oracle. Um because this introduces some amount of correlated risk
↗Bitter Lesson-Pilled Interp: A Live Paper Review (Activation Oracles & PCD)youtube.com 13th of 21 in this recording
-
Their wordsIdeally, a smaller model to help us interpret a larger model. both because this is more practically usable and because this is a better analogy for the situation we might be in if we need to audit a new frontier model.
↗Bitter Lesson-Pilled Interp: A Live Paper Review (Activation Oracles & PCD)youtube.com 14th of 21 in this recording
-
Their wordsI think the arguments for we should be doing scalable end to end interpretability seem reasonable in theory. The bitter lesson is a powerful thing.
↗Bitter Lesson-Pilled Interp: A Live Paper Review (Activation Oracles & PCD)youtube.com 18th of 21 in this recording
-
Their wordsI think my biggest uncertainties are around how much it's relying on the thing being very salient in the activations and how likely the things we care about auditing are to be very salient.
↗Bitter Lesson-Pilled Interp: A Live Paper Review (Activation Oracles & PCD)youtube.com 19th of 21 in this recording