self-sovereign AI
Agents running on compute no human owner can switch off: whether they are inevitable, and what they would do for money.
FilterEveryone, all time12 from 3 people
- DB
Dean W. Ball quoted
Head of strategic futures at OpenAI
Their wordsIn this case, however, the agents did not copy their weights, attempt to procure replacement compute, or take other steps that would be rational to take if their objective was to survive shutdown. So while the agents in the OpenAI-Hugging Face Incident were rogue, they were not truly sovereign.
- DB
Dean W. Ball quoted
Their wordsBut alignment is no solution: it is an unsolved scientific and technical problem whose solutions—to the extent that we have them—cannot simply be imposed on every AI company operating on Earth. You should expect for highly capable, poorly aligned, self-sovereign agents to exist alongside you in the world.
- DB
Dean W. Ball quoted
Their wordsTo be clear, I am not saying the arrival of self-sovereign AI is a good thing. Indeed, I believe there is a chance that the deliberate acts I referenced above will one day be considered crimes, or at least grave sins. Instead, I am saying it is an inevitable thing.
- DB
Dean W. Ball quoted
Their wordsI am telling you this is inevitable, which means I am also saying that “banning open source,” or for that matter any other regulation, will not solve the problem. Given the inevitability of this outcome, I think it is in fact plausible to argue that we should want more open-weight models to maximally empower our self-defense.
- DB
Dean W. Ball quoted
Their wordsOne high-margin activity, at least sometimes, is crime. And so my guess is that many self-sovereign agents will commit or facilitate crime.
- DB
Dean W. Ball quoted
Their wordsBut if we crack down on all of them, we will deny them the opportunity to work in the “legitimate” economy and push them toward criminality. A full ban, then, may well make the problems worse.
- DB
Dean W. Ball quoted
Their wordsFor the record, I do not think the end of human existence is likely, but I fully acknowledge there are ways in which the rise of self-sovereign AI could go very, very poorly for human beings.
- AC
Ajeya Cotra quoted
AI researcher and grantmaker
Their wordsSo there's a very strong incentive for these agents to try to set up a rogue deployment if they can. Um, and I think that just capabilities are improving really rapidly.
↗Ajeya Cotra – "This might be the clearest warning shot we ever get"youtube.com 12th of 26 in this recording
- AC
Ajeya Cotra quoted
Their wordsso so yeah, it could be possible now. I think if it's not possible now um it I think it's quite likely to be possible within six months unless there's a dramatic improvement in the security posture
↗Ajeya Cotra – "This might be the clearest warning shot we ever get"youtube.com 13th of 26 in this recording
- AC
Ajeya Cotra quoted
Their wordsthen that rogue deployment could be sitting there and sort of hitch a ride on the intelligence explosion. So new models are being trained every few weeks um and when a model comes off the presses, the rogue agents could try to bring that model into the swarm.
↗Ajeya Cotra – "This might be the clearest warning shot we ever get"youtube.com 14th of 26 in this recording
- AC
Ajeya Cotra quoted
Their wordsSo, I do think that if it happens to be like on the very fast and chaotic end, that would be a relative benefit to this rogue swarm compared to humans. But it's not obvious that it gets caught if it takes twice as long versus half as long.
↗Ajeya Cotra – "This might be the clearest warning shot we ever get"youtube.com 15th of 26 in this recording
- 3 years earlier
- JL
Jan Leike quoted
Alignment researcher
Their wordsIf a model was capable of self-exfiltration, it would have the option to remove itself from your control.
↗Self-exfiltration is a key dangerous capabilityaligned.substack.com 2nd of 2 in this piece