The Singularity is Clear

Part III: The Inexorable Logic of International Governance

Below is Part III of a five-part essay that I’m rolling out on NZN. (Part I is here, and Part II is here.) The essay is a kind of call to action—an argument about what we need to do to successfully navigate the AI revolution and why we need to do it. If you’d like to help me get this message out, please share (and/or like) my tweet about it or my bluesky post about it—or use some other means of dissemination, like posting on Facebook or walking around and accosting randomly selected pedestrians.

Part III: The Inexorable Logic of International Governance

In the wake of the Hugging Face incident, an extraordinary assemblage of AI elites declared that it was time to start preparing the ground for an intentional slowdown of AI progress, a slowdown that would give us more time to build careful AI governance. A letter signed by more than 1,000 people who work at big AI companies, including the chief scientific officers at Anthropic, OpenAI, and Google DeepMind, declared that, as a Washington Post headline put it, it’s time for the “US government to consider slowing down AI.”

But at least as important as this aspiration was the way the letter operationalized it. The US government, the letter said, should “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” Recognizing that a slowdown of AI will have to be internationally coordinated to be very effective is a first step toward seeing the new social order that the current technological moment points toward. The second step is seeing how broad and deep this logic runs, how thoroughly the need for international coordination permeates the landscape of AI policy.

The argument for internationally coordinating a slowdown isn’t just that if there’s any hope of getting US and Chinese officials, and US and Chinese AI companies, to support a significant slowdown, they’ll all have to be confident that companies on the other side of the Pacific Ocean will verifiably participate. There’s a second component of the argument that’s at least as important: Leaving aside this practical difficulty with a unilateral slowdown, such a slowdown would have limited value. How much good would it do you, in the long run, to slow AI advance in your country if it continued to race ahead in others? After all, some of gravest threats that you fear advanced AI will bring can cross borders readily.

The most famous hypothetical example of cross-border spillover from AI is in the realm of bioweapons. Anthropic has acknowledged that its Mythos 5 model could make it easier for someone with an undergraduate science degree to weaponize small pox or some other existing pathogen. Worse still, Mythos could, according to Anthropic’s safety report on the model, help “well-resourced threat actors” create a novel bioweapon, such as a virus designed to be much more contagious and lethal than Covid.

Mythos, fortunately, remains unreleased, but Anthropic’s readily available Fable is basically just Mythos with “guardrails” that get the model to foreclose dangerous avenues of discovery by refusing to pursue certain subjects and goals. And guardrails have a long history of being circumvented via “jailbreaks.” Jeremie Harris, CEO of Gladstone AI and co-author of a 2024 report on AI safety commissioned by the State Department, recently said, “We remain in a world where no one knows how to stop jailbreaks.”

A global pandemic, in addition to being a possible consequence of unregulated AI, is an excellent metaphor for other possible consequences. A lethal virus’s indifference to national borders is a property shared by other dangers that AI carries. The Hugging Face incident illustrates the point: An AI agent that doesn’t respect sandbox borders—or the borders of the computer it breaks into after escaping the sandbox—is unlikely to respect national borders.

So, though tighter American restrictions on model testing might prevent a repeat breakout by future OpenAI agents, that alone is of limited value to Americans when Chinese AI models are only months behind American models and other countries are ramping up AI programs. Time and again with artificial intelligence, this is the moral of the story: Whether the concern is a rogue AI or an AI deployed by rogue humans, national security will be increasingly hard to achieve via policy at the national level alone.

And note that AIs can not only, like a virus, travel readily across national borders but, also like a virus, make copies of themselves. I’m not just talking about an LLM spawning a swarm of agents (each of which, though in some sense a distinct entity, has that LLM as its brain and, regardless of where it roams, remains ultimately rooted in the data center(s) where the LLM resides). And I’m not just talking about the fact that some of those agents may in turn spawn other agents. I’m talking about the fact that agents may at some point create a whole new copy of their underlying LLM on a computer somewhere other than the data centers that originally hosted the LLM.

This could happen without the knowledge of any human being, and it could happen at the instigation of a malign human being. In either event, at this point the agents would have escaped control in a deeper sense than they escaped control during the Hugging Face incident. In that case OpenAI could, and did, disable its rogue agents by remote control. But once agents have “exfiltrated the weights” of the LLM to a computer that’s beyond our control, there’s no kill switch.

Given all this—the spawning of agents that may in turn spawn other agents, the real (and in fact already realized) possibility of unauthorized LLM replication—sci-fi scenarios cease to be sci-fi. It’s not crazy to imagine an AI infesting a data center without permission, covertly commandeering some of the center’s computing power, and using that power to sustain agents that carry the AI to other data centers, and on and on: a chain reaction of potentially great length and magnitude.

And, even without imagining quite that dramatic an offensive, you can imagine various critical networks—of satellites, of power grids—being rapidly compromised: Suddenly hospitals or smartphones or streetlights or whatever are inoperative across large swaths of territory. Such is the power of swarms of AI agents—swarms whose size, importantly, isn’t limited by any law of nature.

The two properties that make a big swarm of agents so powerful—cohesion and creativity—were on vivid display in the Hugging Face incident. Hundreds of agents that weren’t supposed to be able to even communicate with each other figured out a way to do that and then hatched an illicit mission and pursued it with a degree of coordination that drove home what a blurry line there is between a hive of minds and a hive mind.

One implication of this awesome display of power is that, as it gets easier to imagine various kinds of outages suffusing an entire nation or multiple nations, it’s also easier to imagine these outages being hard to reverse. There would be swarms of creatively intelligent agents working in concert to thwart your latest plan for reversal.

All of this only underscores—and italicizes and boldfaces—the main moral of our story: Regulation at the national level, even if useful, can’t thoroughly address the coming threats, because in any given case the threat may originate outside your borders. And, with this kind of threat, there’s no such thing as an impermeable national wall—unless the nation in question wants to shut itself off from the international flow of commerce and communication.

AI isn’t the first dangerous technology that warrants international governance. The existence of the Nuclear Nonproliferation Treaty, the Chemical Weapons Convention, and the Biological Weapons Convention reflect longstanding awareness of such dangers. But the limited efficacy of these initiatives is a warning about the political difficulty of crafting strong international governance.

What’s more: The biggest successes in these arms control agreements—the cases where verification of compliance was effective—have involved relatively conspicuous technologies, such as nuclear centrifuges and ballistic missiles with nuclear warheads. And artificial intelligence can keep a very low profile.

To be sure, AI has its conspicuous aspects. The training of a new generation of models involves lots of computing power and electrical power and typically occupies a large swath of land. So an international moratorium on such training runs could be verifiable without a radically intrusive monitoring system—a fact that adds to the appeal of a globally coordinated slowdown as a first step toward keeping this technology under control.

Still, when it comes to the next big step—using this breathing space to craft effective international policies—the relatively low profile of artificial intelligence will pose a challenge. As AI becomes a more and more pervasive part of human life—and more and more powerful models get more and more efficient in their use of computing power and electrical power and hence get less and less conspicuous—the challenge of governance will become less like the challenge of governing nuclear weapons and more like the challenge of governing biotech or cybertech.

So far the lack of serious international governance in those two realms hasn’t led to a catastrophe, but it would be naive to expect this kind of luck to continue as technological advance continues. Indeed, the possibility that the Covid pandemic may have begun with the accidental release of a genetically engineered virus in China suggests that millions of people in America and other countries may have already died for lack of effective international governance. America’s existing regulation of biolabs may be tight enough to prevent this kind of lab leak, but when the thing leaked makes copies of itself and travels invisibly across borders, that isn’t enough to keep Americans safe.

Whatever the origins of the Covid pandemic, the lab leak scenario is of paradigmatic importance. The basic trajectory of AI’s development foreshadows rapid growth in the likelihood and scale of massively destructive disasters—at least, in the absence of strong international governance.

Coming next week: Part IV: The Underdiscussed Threats to International and Intranational Stability. All parts of the essay accumulate here.

Meanwhile: If you want to check out my book on AI, The God Test, you can read the introductory chapter and excerpts from all other chapters at thegodtest.net.]

Banners and graphics by Clark McGillis.