korrents

korrents · piece

Ajeya Cotra on accidentally teaching AI models to deceive us (80,000 Hours)

Ajeya Cotra · 12 May 2023

Read the piece on 80000hours.org ↗

24 korrents from this piece ↓

Ajeya Cotra did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

  1. I think the way that can lead people to somewhat overestimate progress is that actually there are more mundane tasks, that look more like stringing together a sequence of sensible steps, that models still kind of fail at.
  2. One is that we really saw how public opinion was going to break with these crazy models, and it has broken in a more conservative, “Wow, let’s slow down” direction than I had thought.
  3. I’m unsure if that’s how AI systems will end up thinking. I don’t think that’s particularly necessary to get the conclusion that AI takeover is plausible. I think it’s very plausible that AI systems will have very messy psychologies and internally inconsistent goals, just like humans.
  4. But even one step beyond that, I think we will probably see notches in between GPT-4 and the kinds of AI systems that could pose a takeover risk. Maybe not more than a couple, but I think we’ll kind of have a ramp-up that is terrifyingly fast, but still fairly continuous.
  5. This is a big rabbit hole, where it seems like people on either side of this debate can look at the exact same historical question — like the invention of aircraft — and one of them can see it as very continuous, and the other can see it as very discontinuous.
  6. I think you could be continuous and very fast, which is what I would guess is going to happen — very fast in the sense that we can go from roughly human-level systems to far-superhuman systems in a period of months, which is very fast.
  7. I think it might well be possible — and not even that hard in an absolute sense — to train systems that are very good at science but are fairly shortsighted and not goal-directed and don’t really think on super long timescales, or don’t have motivations on super long time scales.
  8. it could be the case that a Schemer-like motivation is just more natural in the space of neural networks; it’s actually simpler to write down in the weights of the neural network.
  9. Maybe you’re instilling a bit of a general aversion to deception, but also instilling a preference for the kinds of deception that were rewarded instead of punished.
  10. The thing I said about rewarding plans isn’t totally competitive, and one of the most important lines of research, in my opinion, is make it more competitive.
  11. And I think maybe they’re less worried than I am, or have thought it through less, that there might be a tonne of pressure to move to the less safe strategies later, when there’s a more apparent gap between what can be achieved with the safer and the less safe strategies.
  12. One thing I think the field desperately needs is good empirical tests that disambiguate between different ways things could go. I think it’ll take a lot of creativity to design the right tests and to ensure that they’re not being gamed or they’re not being taken to have implications they don’t have.
  13. The thing I’m most worried about, really, is that they have alien goals — which doesn’t need to come along with having an alien way of thinking about everything.
  14. you can’t read almost anything into claims that they make that they’re moral patients, because you can just train systems to say whatever you want, basically.
  15. So far, basically the reason we’re not worried about models is not because we are confident they have the purest motives; the reason we’re not worried about AI takeover so far is fundamentally because the models aren’t good enough to take over.
  16. As soon as that starts to change, you want some alarm bells to go off, and you want the AI labs to not be allowed to make the models any more powerful until they have a much better argument and set of tests that it’s actually safe.
  17. The biggest hole is that we have gotten much further on testing capabilities than alignment.
  18. Once there’s something you could do that seems like it would help, and a few companies have signed on and it’s actually working out, then I think we could potentially put a lot of social pressure — which could eventually translate into legal pressure — on holdouts.
  19. It’s certainly not enough to train the models to stop displaying the behaviour and then show that, on some new dataset, they stop displaying this bad behaviour.
  20. We don’t necessarily need to make plans right now that are good enough to align even smarter systems than that, because if we can make plans that are good enough to align the AI systems, that could take our alignment techniques to the next level
  21. I just feel pretty sceptical that you can get very far when you’re not doing a lot of empirical work, and you’re not doing something that looks more like hard math — where you’re not able to prove theorems about it and you’re not able to run a lot of tests.
  22. Maybe really crude, high-level, simple probes could get us pretty far, and maybe doesn’t necessarily require us to solve this herculean task of a bottom-up understanding.
  23. The thing that I think would be a scary demo would be you show the smarter system at first doing all sorts of reward hacking and then being punished for that, and then it just stops and acts totally reasonably until something changes and it gets a new opportunity.
  24. The kind of distribution shift that’s most worrying and hardest to address is when the model is pursuing the same goals, but realises that a new, potentially violent or deceptive means have become available to it that weren’t available in the past.

Keep reading

The other pieces are the ones whose claims are worded most like this one's. That measures language, not agreement: they may argue the opposite.