korrents
Neel Nanda

What Neel Nanda thinks about LLMs

@neel-nanda · 65 positions · On ppll ↗

Runs the mechanistic interpretability team at Google DeepMind; worked on interpretability at Anthropic before that.

Neel Nanda did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

14 dated positions, 2025, in their own words. Our reading of what Neel Nanda has said — not written or endorsed by them.

  1. benchmarks

    I think that the like standard paradigm of evaluation where you just have some static benchmark um maybe a benchmark with like an LLM judge at the end with a specific rubric is kind of sus and not something that I would be fully happy with in the long term as methods of evaluating frontier systems

    ↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 1st of 27 in this recording

  2. And yeah, it's great that they checked and reported this prominently because, you know, a really easy way to make a model incredibly safe is to replace it with a rock that always says, "As a large language model, I cannot help you with this."

    ↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 5th of 27 in this recording

  3. scaling laws

    And they have like a really strong prior for continuing sentences because you know if you don't continue a sentence properly you normally fail in pre-training which is a strong incentive to really deeply ingrain the reflex to continue sentences.

    ↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 6th of 27 in this recording

  4. typically models currently seem not very good at hiding things in the chain of thought. So the hope is that they are just kind of bad at things like like if they're being deceptive, it will be apparent in the chain of thought.

    ↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 8th of 27 in this recording

  5. I would generally say you should expect these numbers to be underestimates of the eval awareness, especially for more capable models which are going to be more capable of kind of holding things in their head and not saying them.

    ↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 11th of 27 in this recording

  6. AI alignment

    like we don't want to reinforce deception in models. That seems bad. That seems the kind of thing that could cause misaligned behavior in general. And thus, if we are training them to do role playing, this plausibly causes badness.

    ↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 15th of 27 in this recording

  7. but if you're using data generated by other models and those models had certain flaws, this could easily get instilled into your model.

    ↗How Aligned Is Claude? A Live Review of the Opus 4.5 System Cardyoutube.com 17th of 27 in this recording

  8. 7 days earlier
  9. there's a good amount of work from me and others showing that when models receive entities in the prompt, they often look up everything they know about that entity on the name of that entity, not just whenever they actually need that information.

    ↗Bitter Lesson-Pilled Interp: A Live Paper Review (Activation Oracles & PCD)youtube.com 4th of 21 in this recording

  10. one of my pet hypotheses is that um the more uncertainty there is about what comes after a token um the more useful it is to just represent all of the relevant info from the context

    ↗Bitter Lesson-Pilled Interp: A Live Paper Review (Activation Oracles & PCD)youtube.com 7th of 21 in this recording

  11. Though from another perspective, I think that auditing things from a model that don't shape its actions is like probably doomed. weight-based interpretability has not been going that well in my opinion

    ↗Bitter Lesson-Pilled Interp: A Live Paper Review (Activation Oracles & PCD)youtube.com 10th of 21 in this recording

  12. AI alignment

    I think another thing that I'm a little bit sketched out by is using the model itself as the activation oracle. Um because this introduces some amount of correlated risk

    ↗Bitter Lesson-Pilled Interp: A Live Paper Review (Activation Oracles & PCD)youtube.com 13th of 21 in this recording

  13. Ideally, a smaller model to help us interpret a larger model. both because this is more practically usable and because this is a better analogy for the situation we might be in if we need to audit a new frontier model.

    ↗Bitter Lesson-Pilled Interp: A Live Paper Review (Activation Oracles & PCD)youtube.com 14th of 21 in this recording

  14. And yes, I strongly agree that figuring out what applications benefit from like things fancier than just asking the model or really simple crude interpretability techniques like um prefill attacks would be great.

    ↗Bitter Lesson-Pilled Interp: A Live Paper Review (Activation Oracles & PCD)youtube.com 15th of 21 in this recording

  15. And then these are a massive pain to train such that realistically the resources required to use them on a frontier model are just not worth it.

    ↗Bitter Lesson-Pilled Interp: A Live Paper Review (Activation Oracles & PCD)youtube.com 17th of 21 in this recording