11 korrents from this piece
Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.
Our reading
Their wordsIn this case, however, the agents did not copy their weights, attempt to procure replacement compute, or take other steps that would be rational to take if their objective was to survive shutdown. So while the agents in the OpenAI-Hugging Face Incident were rogue, they were not truly sovereign.
Our reading
Their wordsBut alignment is no solution: it is an unsolved scientific and technical problem whose solutions—to the extent that we have them—cannot simply be imposed on every AI company operating on Earth. You should expect for highly capable, poorly aligned, self-sovereign agents to exist alongside you in the world.
Our reading
Their wordsTo be clear, I am not saying the arrival of self-sovereign AI is a good thing. Indeed, I believe there is a chance that the deliberate acts I referenced above will one day be considered crimes, or at least grave sins. Instead, I am saying it is an inevitable thing.
Our reading
Their wordsI am telling you this is inevitable, which means I am also saying that “banning open source,” or for that matter any other regulation, will not solve the problem. Given the inevitability of this outcome, I think it is in fact plausible to argue that we should want more open-weight models to maximally empower our self-defense.
Our reading
Their wordsThis is the sole intrinsic thing about AI that prevents agents from truly infinite self-replication. They will be constrained by the need to find and pay for sufficient compute to run themselves.
Our reading
Their wordsOne high-margin activity, at least sometimes, is crime. And so my guess is that many self-sovereign agents will commit or facilitate crime.
Our reading
Their wordsBut if we crack down on all of them, we will deny them the opportunity to work in the “legitimate” economy and push them toward criminality. A full ban, then, may well make the problems worse.
Our reading
Their wordsAgents will need persistent, unique identifiers that allow their actions to be traced back to a responsible actor. Doing this successfully will also require human users to possess a unique identifier.
Our reading
Their wordsBut a society that does not have anonymous speech does not really have genuine freedom of expression, and we should embed this principle into any digital identification system we attempt.
Our reading
Their wordsSo many of the people who think about the governance of superintelligence, myself included, avoided that unpleasantness and bowed to the social pressure to self-censor.
Our reading
Their wordsFor the record, I do not think the end of human existence is likely, but I fully acknowledge there are ways in which the rise of self-sovereign AI could go very, very poorly for human beings.