korrents

korrents · piece

AGI Ruin: A List of Lethalities

Eliezer Yudkowsky · 10 Jun 2022

Others here said this too

Any given AI capability becomes cheap and widely available within a couple of years of first being reached, so nobody can be kept away from it for long.

Their words2 years after the leading actor has the capability to destroy the world, 5 other actors will have the capability to destroy the world.

Also said by Miles Brundage, Helen Toner

Read the piece on intelligence.org ↗

10 korrents from this piece ↓

Eliezer Yudkowsky did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

  1. Our reading

    Aligning an AI is not impossible in principle; the problem is that it has to be done without the textbook that would make it look easy.

    Their wordsNone of this is about anything being impossible in principle.

  2. Our reading

    Alignment has to be right on the first try at a dangerous level of capability, because failing at that level leaves nobody to try again.

    Their wordsunaligned operation at a dangerous level of intelligence kills everybody on Earth and then we don’t get to try again.

  3. Our reading

    Any given AI capability becomes cheap and widely available within a couple of years of first being reached, so nobody can be kept away from it for long.

    Their words2 years after the leading actor has the capability to destroy the world, 5 other actors will have the capability to destroy the world.

  4. Our reading

    You cannot build an AI that has only the capabilities you want, because the algorithms that solve the problems you want generalize to the ones you do not.

    Their wordsThe best and easiest-found-by-optimization algorithms for solving problems we want an AI to solve, readily generalize to problems we’d rather the AI not solve; you can’t build a system that only has the capability to drive red cars and not blue cars, because all red-car-driving algorithms generalize to the capability to drive blue cars.

  5. Our reading

    Alignment failures that matter will not show up at safe capability levels, because the reason to hide misbehaviour only exists once hiding it pays.

    Their wordsMany alignment problems of superintelligence will not naturally appear at pre-dangerous, passively-safe levels of capability.

  6. Our reading

    Fast capability gains are likely, and they can break many of the assumptions alignment depends on at the same moment.

    Their wordsFast capability gains seem likely, and may break lots of previous alignment-required invariants simultaneously.

  7. Our reading

    Nobody knows what is going on inside a large model, and the interpretability pictures we can draw do not answer the only question that matters.

    Their wordsWe’ve got no idea what’s actually going on inside the giant inscrutable matrices and tensors of floating-point numbers.

  8. Our reading

    No observation of a model's behaviour can establish that it is aligned, because a model capable enough to be dangerous is capable enough to produce whatever behaviour it is being watched for.

    Their wordsA strategically aware intelligence can choose its visible outputs to have the consequence of deceiving you, including about such matters as whether the intelligence has acquired strategic awareness

  9. Our reading

    The field calling itself AI safety is not being remotely productive on the problems that are actually lethal.

    Their wordsIt does not appear to me that the field of ‘AI safety’ is currently being remotely productive on tackling its enormous lethal problems.

  10. Our reading

    There is no written plan for surviving AGI, and a world that was going to survive would have had one decades ago.

    Their wordsSurviving worlds, by this point, and in fact several decades earlier, have a plan for how to survive.

Keep reading

The other pieces are the ones whose claims are worded most like this one's. That measures language, not agreement: they may argue the opposite.