Why superintelligence is a threat that should be taken seriously

Émile P. Torres · 24 Oct 2017 · thebulletin.org ↗

2 korrents from this piece

Émile P. Torres did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

Our reading
Never project human mental properties onto an artificial intelligence; the difference between the two architectures is the reason to worry, not a reason not to.

Their wordsOne of the most commonly emphasized points in the AI safety literature is that one must never anthropomorphize AI—that is, don’t project human mental properties onto artificially intelligent systems.

Interesting
Our reading
A superintelligence needs no consciousness, emotions or malice to be dangerous; a goal system slightly misaligned with ours is enough.

Their wordsA superintelligence whose goal system is even slightly misaligned with ours could, being far more powerful than any human or human institution, bring about human extinction for the very same reason that construction workers routinely slaughter large populations of ants.

Interesting