korrents

korrents · video

Bitter Lesson-Pilled Interp: A Live Paper Review (Activation Oracles & PCD)

Neel Nanda · 3h 03m · youtube.com ↗

21 korrents from this recording

1h2h3h
Neel Nanda did not write this page.

Every claim below is a statement made in this recording, quoted word for word and linked to the second it was said, so you can hear it rather than take our word for it. The wording comes from the transcript published alongside the recording; the sentence above each quote is our reading of the claim, not their wording.

  1. 0:01:21 · watch on youtube.com↗

    Being smart is not the most important thing. You need to be scalable and you need to be good at flexibly using large amounts of compute to do the things that you care about doing.
  2. 40 min later
  3. 0:41:26 · watch on youtube.com↗

    Um, so I think that interpretability agents are actually really nice for this kind of thing because the interpretability agents are reproducible in a way that human researchers aren't.
  4. 12 min later
  5. 0:53:33 · watch on youtube.com↗

    I would weakly guess that agents capabilities are getting better at a faster rate than a model's ability to be a good activation oracle. Though that is a wild hot take
  6. 5 min later
  7. 0:59:00 · watch on youtube.com↗

    there's a good amount of work from me and others showing that when models receive entities in the prompt, they often look up everything they know about that entity on the name of that entity, not just whenever they actually need that information.
  8. 5 min later
  9. 1:04:13 · watch on youtube.com↗

    Linear probing is OP. Um and attention head probes are even more OP, but whatever.
  10. 1:04:27 · watch on youtube.com↗

    I endorse doing easy to do things that will probably work just in case they fail because if they fail, that is info.
  11. 7 min later
  12. 1:11:34 · watch on youtube.com↗

    one of my pet hypotheses is that um the more uncertainty there is about what comes after a token um the more useful it is to just represent all of the relevant info from the context
  13. 2 min later
  14. 1:13:36 · watch on youtube.com↗

    And I would basically say you should expect it to be in the activations when it is necessary or like when it has a causal impact on the actions the model is taking.
  15. 1 min later
  16. 1:14:55 · watch on youtube.com↗

    I think this is a weakness of the fine-tuned model organisms studied in that because they are fine-tuned, there's no downside to not think about the fine-tuning objective all the time
  17. 1 min later
  18. 1:15:54 · watch on youtube.com↗

    Though from another perspective, I think that auditing things from a model that don't shape its actions is like probably doomed. weight-based interpretability has not been going that well in my opinion
  19. 10 min later
  20. 1:25:35 · watch on youtube.com↗

    I want a bunch of decorrelated techniques that can all provide a consistent picture because I just don't really see any one technique getting to the point of reliability where it alone could justify something dramatic like this model is not safe enough to use.
  21. 1 min later
  22. 1:26:58 · watch on youtube.com↗

    a depressing lack of good evidence of things that were done by analyzing mechanisms that could not have been done by just analyzing a model's activations.
  23. 1 min later
  24. 1:27:57 · watch on youtube.com↗

    I think another thing that I'm a little bit sketched out by is using the model itself as the activation oracle. Um because this introduces some amount of correlated risk
  25. 3 min later
  26. 1:30:34 · watch on youtube.com↗

    Ideally, a smaller model to help us interpret a larger model. both because this is more practically usable and because this is a better analogy for the situation we might be in if we need to audit a new frontier model.
  27. 2 min later
  28. 1:32:52 · watch on youtube.com↗

    And yes, I strongly agree that figuring out what applications benefit from like things fancier than just asking the model or really simple crude interpretability techniques like um prefill attacks would be great.
  29. 6 min later
  30. 1:38:44 · watch on youtube.com↗

    Like the fact that it generalizes to new tasks is in my opinion the central result of this paper. Um, that suggests this is actually a method with promise.
  31. 1 min later
  32. 1:39:38 · watch on youtube.com↗

    And then these are a massive pain to train such that realistically the resources required to use them on a frontier model are just not worth it.
  33. 18 min later
  34. 1:57:20 · watch on youtube.com↗

    I think the arguments for we should be doing scalable end to end interpretability seem reasonable in theory. The bitter lesson is a powerful thing.
  35. 3 min later
  36. 2:00:40 · watch on youtube.com↗

    I think my biggest uncertainties are around how much it's relying on the thing being very salient in the activations and how likely the things we care about auditing are to be very salient.
  37. 16 min later
  38. 2:16:29 · watch on youtube.com↗

    And presumably if you train it a bunch longer, things might change in further weird qualitative ways. And yeah, this is like pretty interesting and suggests that it might be hard to form confident takes from cheap small scale experiments.
  39. 37 min later
  40. 2:53:06 · watch on youtube.com↗

    And I think the interesting question is like the longevity of the technique. Like where could it be in like a year not where is it now?