What Neel Nanda thinks about reinforcement learning
Runs the mechanistic interpretability team at Google DeepMind; worked on interpretability at Anthropic before that.
Neel Nanda did not write this page.
We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.
Filter reinforcement learning · Newest first
Only Neel Nanda on reinforcement learning. See everyone on reinforcement learning
4 dated positions, 2025, in their own words. Our reading of what Neel Nanda has said — not written or endorsed by them.
-
Their wordsUm, more generally, I think that um, the easier version of this of just after a post- training run, go look at the RL transcripts and figure out problems with the various environments use and see if you can use this for debugging things. Seems like a pretty good thing to do in general.
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 2nd of 27 in this recording
-
Their wordsit seems likely that a bunch of alignment relevant properties will be introduced during RL training. Um, not we should not assume this will be true for all such properties but like it seems plausible especially for the things like goal seeking, reward hacking, things like that.
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 3rd of 27 in this recording
-
Our readingHow aligned current models are depends a lot on things like their degree of reward hacking.
Their wordsI kind of feel like a lot of what affects how aligned I think current models are are like things like their degree of reward hacking
↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 27th of 27 in this recording
- 7 days earlier
-
Their wordsI think this is a weakness of the fine-tuned model organisms studied in that because they are fine-tuned, there's no downside to not think about the fine-tuning objective all the time
↗Bitter Lesson-Pilled Interp: A Live Paper Review (Activation Oracles & PCD)youtube.com 9th of 21 in this recording