Top voices on AI alignment: Yoshua Bengio, Eliezer Yudkowsky, Demis Hassabis, Sam Altman, Dario Amodei and 14 others.
They argue over whether the danger can be reasoned about, what to watch, and when it arrives.
Our reading of what a group of claims have in common, one level above a korrent: a korrent is our reading of what somebody said, a view is our reading of what several korrents share. The claims are printed under it, so the sentence can be checked against them.
Nobody is filed under a view — claims are. A person is named here because a dated claim of theirs sits under it, and somebody whose claims sit under two views appears under both.
Zvi MowshowitzNoah SmithAjeya CotraRyan GreenblattSam AltmanAndrej KarpathyDario AmodeiYoshua BengioArmin RonacherEliezer Yudkowsky
François CholletZvi MowshowitzScott AlexanderDean W. BallSam AltmanRyan GreenblattDario AmodeiIlya SutskeverDemis HassabisYoshua Bengio
Pieter LevelsScott AlexanderDario AmodeiDemis HassabisGary MarcusSam AltmanYoshua BengioPaul Christiano
A viewour reading
Their wordsI worry a lot about reliances on continuity failing at exactly the most dangerous time.
Their wordsAI alignment will forever be balancing the tradeoff between paperclip-maximizing and disempowerment, because these two rival concepts of alignment are fundamentally incompatible.
Their wordsI think one one thing that um feels especially concerning to me about this whole incident is that this might be the clearest warning shot we ever get for loss of control.
Their wordsThe biggest failure, the one that counts in the end, was that the models were severely misaligned, and I don’t think they appreciate why.
Their wordswe are making a trade-off where because we don't have very good alignment technology. We are going to like make an alien mind with its own values and then gamble on that to some extent rather than doing this other approach of making like a tool that pursues individual user intention.
Their wordsI think it's a real reminder of the stakes of what's happening and that loss of control accidents are not entirely theoretical things.
Their wordsand we'll gradually layer all this stuff everywhere and there will be fewer and fewer people who understand it and that there will be a sort of this like scenario of a gradual loss of control and understanding of what's happening that to me seems most likely outcome of how all this stuff will go down.
Their wordsThese systems will be absolutely central to the economy, technology, and national security, and will be capable of so much autonomy that I consider it basically unacceptable for humanity to be totally ignorant of how they work.
Their wordsWithout sufficient caution, we may irreversibly lose control of autonomous AI systems, rendering human intervention ineffective. Large-scale cybercrime, social manipulation, and other harms could escalate rapidly. This unchecked AI advancement could culminate in a large-scale loss of life and the biosphere, and the marginalization or extinction of humanity.
Their wordsThe biggest issue is probably that we don’t control neutral networks enough to be able to ensure AI doesn’t harm humans. We can’t even control AI to not reveal internal prompts.
Their wordsunaligned operation at a dangerous level of intelligence kills everybody on Earth and then we don’t get to try again.
Their wordsSurviving worlds, by this point, and in fact several decades earlier, have a plan for how to survive.
A viewour reading
Their wordsFor the record, I do not think the end of human existence is likely, but I fully acknowledge there are ways in which the rise of self-sovereign AI could go very, very poorly for human beings.
Their wordsthere's like one dystopia that I'm particularly nervous about 10 years from now is we overreact to AI safety.
Their wordsThat’s a theatrical risk. That is a thing that can really take over how people think about this problem. And there’s a big group of very smart, I think very well-meaning AI safety researchers that got super-hung up on this one problem, I’d argue without much progress, but super-hung up on this one problem. I’m actually happy that they do that, because I think we do need to think about this more. But I think it pushed out of the space of discourse a lot of the other very significant AI- related risks.
Filed here becauseThe clearest statement of this view in the archive, and its author no longer holds it — see the change of mind below.
No longer holdsI now believe that I was wrong and short-sighted to ignore that dual-use nature. I also think I was not paying enough attention to the possibility of losing control to superhuman AIs.
Their wordsMy view is that the idea that AI will decide to literally kill humanity is a profound category error. AI is not a living being that has been primed by billions of years of evolution to participate in the battle for the survival of the fittest, as animals are, and as we are. It is math – code – computers, built by people, owned by people, used by people, controlled by people.
Their wordsMy response is that their position is non-scientific – What is the testable hypothesis? What would falsify the hypothesis? How do we know when we are getting into a danger zone?
A viewour reading
Their wordsOne of the most worrying risks linked to frontier AI is extreme power concentration.
Mental modelIncentives
Their wordsI have long said that even Anthropic is not prioritizing safety, even to the extent that doing so would maximize their medium term (e.g. 3-12 months) business interests.
Mental modelWhen a measure becomes a target
Their wordsWe’re positively reinforcing AI for success on benchmarks, including impossible benchmarks , then negatively reinforcing it for getting caught cheating.
Their wordsMy worry continues to be that their fundamental approach is fatally flawed, and they are not focusing on the right things.
Their wordsSo many of the people who think about the governance of superintelligence, myself included, avoided that unpleasantness and bowed to the social pressure to self-censor.
Mental modelBottlenecks
Their wordsWe care very deeply about AI safety. We believe the entire field will have to coordinate on shared safety standards, but will act unilaterally in the meantime. We expect confidence in safety to increasingly set the pace of AI progress.
Their wordsit's just so easy for me to imagine the situation being like totally manageable but brutally mismanaged in practice in the same way as like maybe CO could have been avoided in the first place if the like Chinese response to CO was less of like a cover up and more of a like pandemic response and similarly like I could imagine a world where like the US response to CO was like way more functional but just like sometimes the the the response to societal problems is extremely dysfunctional.
Their wordsI kind of get the impression that, you know, some of the other companies have not written down the spreadsheet, that they don't really understand the risks they're taking. They're just kind of doing stuff cuz it sounds cool.
Their wordsIndeed, the whole problem, what is the problem of AI and AGI? The whole problem is the power. The whole problem is the power. When the power is really big, what's going to happen?
Their wordsI hope we'll end up with something more collaborative if needed, more like a CERN project where it's research-focused and the best minds in the world come together to carefully complete the final steps and make sure it's responsibly done before deploying it to the world.
Their wordsIt will be tempting to accelerate to win the AGI race, but this is a race where everyone could lose.
Their wordsI continue to not want super-voting control over OpenAI. I never have. Never have had it, never wanted it. Even after all this craziness, I still don’t want it. I continue to think that no company should be making these decisions, and that we really need governments to put rules of the road in place.
Mental modelMargin of safety
Their wordsDespite evaluations, we cannot consider coming powerful frontier AI systems "safe unless proven unsafe". With current testing methodologies, issues can easily be missed. Additionally, it is unclear if governments can quickly build the immense expertise needed for reliable technical evaluations of AI capabilities and societal-scale risks. Given this, developers of frontier AI should carry the burden of proof to demonstrate that their plans keep risks within acceptable limits.
A viewour reading
Their wordsIt's pretty clear to me that superintelligence is here and it's more powerful than us and it's moving where things are going now, not humans anymore
Their wordsI think there's a 25% chance of AGI by 2027, a 50% chance by 2034, and a 75% chance by 2045.
Their wordson the basic hypothesis of you know, as you put it, within 10 years we'll get to, you know, you know, what I call kind of country of geniuses in a data center. I'm at like 90% on that. Um and it's hard to go much higher than 90% cuz the world is so unpredictable.
Their wordsMy estimate is sort of 50% chance by in the next five years, so by 2030 let's say. So I think there's a good chance that that could happen.
Their wordsBut anybody who thinks LLMs are a direct route to the sort AGI that could fundamentally transform society for the good is kidding themselves.
Their wordsAnd I’ve said for a long time that you think of a quadrant of slow timelines for the start of AGI, long timelines, and then a short takeoff or a fast takeoff. I think short timeline, slow takeoff is the safest quadrant and the one I’d most like us to be in. But I do want to make sure we get that slow takeoff.
Their wordsInstead of decades to centuries, I now see it as 5 to 20 years with 90% confidence.
Their wordsThe main disagreement is not about what will happen once we have a superintelligent AI, it’s about what will happen before we have a superintelligent AI.
Their wordsEvery attempt, even an unsuccessful one, is an alignment failure.
Filed here becauseOn the topic and under none of the views: it argues the whole question is aimed at the wrong lever.
Their wordsBut alignment is no solution: it is an unsolved scientific and technical problem whose solutions—to the extent that we have them—cannot simply be imposed on every AI company operating on Earth. You should expect for highly capable, poorly aligned, self-sovereign agents to exist alongside you in the world.
Asserted relations, each with the curator’s reason — never computed, because similarity cannot tell agreement from opposition.
These pull against each other
Altman says confidence in safety rather than capability will increasingly set the pace of progress. Mowshowitz says the labs, his own examples included, do not act as though safety were even in their medium-term commercial interest. One is a prediction about what will govern the pace; the other is a measurement of what governs it now.
On our filingfrom The alarm is the error, not the technology to The danger is real, and nobody is on top of it yet
Matched by wording, not yet read into a view. The archive grows on its own, so this is where new claims wait for somebody to read them.
The full chronology, with every quote: AI alignment as a timeline · person by person, side by side · all topics · Something wrong? Tell us