The biggest supporters and critics of artificial intelligence seem to have reached a consensus: AI companies need to stop pushing the technology's frontier until they understand how it will develop.
Any technology as disruptive, or even potentially so, as AI carries immense risks. But the ideas proposed by the AI safety movement cannot meaningfully reduce those risks.
The leaders of most top AI labs have misgivings about their jobs. They believe their companies could provide tremendous benefits, but also that bad AI could cause even worse problems. DeepMind's Demis Hassabis, Anthropic's Dario Amodei, and OpenAI's Sam Altman cosigned a 2023 statement that "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
They have not yet made up their minds about how to prevent their companies from dooming civilization, but the debate is a live one. Last month, thousands of AI company employees signed a letter requesting that Washington "support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development."
Sebastian Mallaby, biographer and longtime confidant of DeepMind's Hassabis, recently wrote in Foreign Affairs, "now we have arrived at a perilous moment—the birth of what amounts to a new species, one that outsmarts humans." He argued that "an AI equivalent to the Nuclear Nonproliferation Treaty," also known as the NPT, is necessary. Washington must "negotiate joint safety principles with China and other powers."
The many-manored socialist Bernie Sanders thinks this is insufficient. The same day Mallaby's article came out, Sanders demanded that OpenAI, Anthropic, and Meta "pause AI development." He threatened, "If you do not take appropriate action now, my colleagues and I in the U.S. Senate will."
Sanders is already attempting further action. Along with Alexandria Ocasio-Cortez, who recently declared that "woke 1," her previous signature movement, "was crazy," Sanders is hawking a bill that would outlaw construction of additional AI computing power in the United States and related exports. The two claim this will "stop a global race to see which country is the first to eliminate hundreds of millions of jobs, or the first to build an AI that destroys the planet."
They have identified important problems. Most frontier AI models are strongest at manipulating code, and their implications for cybersecurity are so profound that Donald Trump swung this summer from a hands-off approach to limiting their release. The automation of manufacturing—also known as the Industrial Revolution—upended nearly every aspect of society, and the automation of some forms of thought could have similarly sweeping effects.
Unfortunately, these solutions will not work.
The major argument against an NPT-style treaty for AI is the NPT's actual record. There were five recognized nuclear-armed states when the treaty was ratified. Since then, four more countries have acquired the bomb, and more could be on the way. An overwhelming majority of South Koreans want to develop nuclear weapons, and Japanese political leaders are openly discussing the role of atomic warfare in Japan's defense.
American power has done far more to prevent the spread of nuclear weapons than clever technical treaties. The NPT created a mechanism for countries to exchange information about peaceful applications of nuclear power. But most American allies stopped short of developing weapons because they believed the United States would defend them. Israel built its arsenal when U.S. support was uncertain, and Seoul and Tokyo are considering the same course for the same reason. States with more malign intentions, such as North Korea and Pakistan, were not deterred by the existence of a treaty.
The pausers, moreover, are already too late. Mallaby warned that "follower labs" could build automated hacking systems on "open weight" models, and "Once someone has access to such a model, no one is able to prevent them from using it." He thought negotiations with China could prevent the spread of open-weight hacking models, but he was mistaken. Chinese hackers have already used that kind of program to hack Taiwanese government websites.
Washington cannot provide the right kinds of guarantees for an NPT-style treaty to work, and it is too late to deter developers from building hacking weapons. Since cyberattacks are already commonplace, no sane leader would give up advanced cybersecurity tools in the hopes that anyone else would punish an attacker.
AOC and Sanders are even more misguided. A leader at the Israeli company that spotted the attack on Taiwan said, "the basic assumption of every government around the globe" should be that they are permanently being cyberattacked. The left-wing answer to this problem is to let other countries surpass America's leading companies rather than to help Americans stay ahead in identifying and fixing vulnerabilities.
Unilateral disarmament does not work, especially for a country already under attack.
