The case for ensuring that powerful AIs are controlled

Buck Shlegeris · 7 May 2024

Others here said this too

No observation of a model's behaviour can establish that it is aligned, because a model capable enough to be dangerous is capable enough to produce whatever behaviour it is being watched for.

Their wordsThe basic problem with evaluating alignment is that no matter what behaviors you observe, you have to worry that your model is just acting that way in order to make you think that it is aligned.

Also said by Eliezer Yudkowsky

Read the piece on redwoodresearch.substack.com ↗

8 korrents from this piece ↓

Buck Shlegeris did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

Our reading
Labs should make powerful models safe to deploy even if those models are misaligned and actively trying to subvert the safeguards.

Their wordsThat is, labs should make sure that the safety measures they apply to their powerful models prevent unacceptably bad outcomes, even if the AIs are misaligned and intentionally try to subvert those safety measures.
Our reading
Controlling a scheming model needs no research breakthrough, which is what makes it the tractable half of the problem today.

Their wordsWe think no fundamental research breakthroughs are required for labs to implement safety measures that meet our standard for AI control for early transformatively useful AIs; we think that meeting our standard would substantially reduce the risks posed by intentional subversion.
Our reading
Catching an AI trying to cause harm should count as a win, because being caught changes the situation more than the attempt cost.

Their wordsSo when evaluating control, we should count catching an AI red-handed as a win condition.
Our reading
Whether a model is controlled can be settled with capability evaluations, which makes control far easier to check than alignment.

Their wordsBecause evaluating control just requires evaluating capabilities, it's far easier to robustly evaluate than alignment.
Our reading
Risk from a scheming model should be managed the way insider security risk is: assume some of them are compromised and stay safe anyway.
Mental modelMargin of safety

Their wordsWe're advocating that companies handle risk from scheming models in a similar way–striving to ensure that they'll be safe even if their alignment efforts fail to prevent models from scheming.
Our reading
A control evaluation that reports under one per cent risk should be read as several per cent, because the evaluation can itself fail.
Mental modelMargin of safety

Their wordsIn particular, suppose that our control evaluations directly estimate a less than 1% chance of catastrophe if our untrusted AIs are scheming, then the actual risk conditional on scheming is probably more like 5% to 20% due to the potential for failures in the evaluation.
Our reading
No observation of a model's behaviour can establish that it is aligned, because a model capable enough to be dangerous is capable enough to produce whatever behaviour it is being watched for.

Their wordsThe basic problem with evaluating alignment is that no matter what behaviors you observe, you have to worry that your model is just acting that way in order to make you think that it is aligned.
Our reading
Control is a fundamentally limited approach that stops working on sufficiently powerful models, so alignment is still needed in the end.

Their wordsAI control (with only black-box techniques) seems like a fundamentally limited approach.

Keep reading

The other pieces are the ones whose claims are worded most like this one's. That measures language, not agreement: they may argue the opposite.