Managing extreme AI risks amid rapid progress (with 24 co-authors)

Yoshua Bengio · 26 Oct 2023 · arxiv.org ↗

4 korrents from this paper

In plain words

AI races toward goal-chasing systems that act alone, risking massive harms and irreversible loss of human control, while safety work and governance lag far behind. It demands one-third of AI budgets for safety research plus fast government rules that tighten automatically as capabilities grow. It is a consensus position paper outlining risks and a tech-plus-oversight plan, drawing lessons from other safety-critical technologies against today's weak response.

Our summary of the paper, not the authors' words — written to be readable without the field's vocabulary, from the stored copy of the paper and nothing else. Drafted with xai:grok-4.5 and checked by a person. The authors' own sentences are the quotes below.

Yoshua Bengio did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

Our reading
Society's response to AI is incommensurate with the possibility of rapid transformative progress that many experts expect, with safety research lagging and governance lacking mechanisms for misuse, recklessness and autonomy.

Their wordsSociety's response, despite promising first steps, is incommensurate with the possibility of rapid, transformative progress that is expected by many experts. AI safety research is lagging. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness, and barely address autonomous systems.

Interesting
Our reading
AI must be reoriented toward safety rather than capabilities alone, as preparation is already behind schedule and the costs of unpreparedness outweigh premature action.

Their wordsFor AI to be a boon, we must reorient; pushing AI capabilities alone is not enough. We are already behind schedule for this reorientation. The scale of the risks means that we need to be proactive, as the costs of being unprepared far outweigh those of premature preparation.

Interesting
Our reading
Without sufficient caution, advanced autonomous AI could cause irreversible loss of human control and culminate in large-scale loss of life or human extinction.

Their wordsWithout sufficient caution, we may irreversibly lose control of autonomous AI systems, rendering human intervention ineffective. Large-scale cybercrime, social manipulation, and other harms could escalate rapidly. This unchecked AI advancement could culminate in a large-scale loss of life and the biosphere, and the marginalization or extinction of humanity.

Interesting
Our reading
Frontier AI systems cannot be presumed safe unless proven unsafe; developers must bear the burden of proof that risks stay within acceptable limits.

Their wordsDespite evaluations, we cannot consider coming powerful frontier AI systems "safe unless proven unsafe". With current testing methodologies, issues can easily be missed. Additionally, it is unclear if governments can quickly build the immense expertise needed for reliable technical evaluations of AI capabilities and societal-scale risks. Given this, developers of frontier AI should carry the burden of proof to demonstrate that their plans keep risks within acceptable limits.

Interesting