Below is the final installment of a five-part essay that I’ve been rolling out here on NZN. (Part I is here, Part II is here, Part III is here, Part IV is here, and the entire essay—all 7,500 words of it!—appears in conveniently unified form here.) The essay is a kind of call to action—an argument about what we need to do to successfully navigate the AI revolution and why we need to do it. If you’d like to help me get this message out, please share (and/or like) my tweet about it or my bluesky post about it—or use some other means of dissemination, such as a swarm of relentless OpenAI agents.
Part V: The Epic Challenge of International Governance
For years the term “warning shot” has had a special meaning within the AI safety community. It refers to a high-profile event that could focus attention on the catastrophic potential of artificial intelligence and thus on the need for preventive measures. Typically, the envisioned event involves significant death and suffering. “Chernobyl-sized” is the way some put it, referring to the 1986 nuclear accident that killed dozens of people in the short run and an estimated tens of thousands via cancer in the long run. Others imagine a higher death toll—maybe millions—as the necessary price for enlightenment about the dangers of AI.
Against that backdrop, some of the most unsettling AI stories of recent months look like good news. We learned in April about the superhacking abilities of Anthropic’s Mythos, and three months later came the Hugging Face story—about the swarm of OpenAI agents that escaped their “sandbox” and then broke various other rules. Neither of these developments, so far as we know, entailed any deaths, and both significantly expanded awareness of AI risk.
Indeed, in the wake of the Mythos story, President Trump—a skeptic of international governance, to say the least—agreed with Chinese leader Xi Jinping to initiate a formal dialogue on AI safety between the two countries. And the aftermath of the Hugging Face incident saw not only the “pacing the frontier” letter, but a subsequent statement by Dario Amodei embracing—in principle at least—the idea of a slowdown.
Still… there’s a ways to go. It’s great that the phrase “international coordination” is now more common in the context of AI, but even the more optimistic observers have modest expectations about the fruits of US-China dialogue. The wildest commonly expressed hope is for the eventual emergence of something comparable to a US-Soviet arms accord during the Cold War—a narrow agreement about specific capabilities that doesn’t involve very intrusive monitoring and doesn’t much alter the fundamentally adversarial nature of the relationship between the two nations.
That would be progress, but not as much progress as the US and China, and the whole world, are going to need in the long run. This technology is only beginning to demonstrate its power—or, more to the point, its powers. Given the number of dimensions along which its capability will grow, and the novel nature of the policy challenges it will pose, preserving international and intranational stability will demand much more thorough and creative policy collaboration between the US and its superpower rival than materialized during the Cold War.
No issue more clearly highlights the degree of innovation in governance that AI will eventually demand than the issue of “open source” (or, strictly speaking, “open weights”) large language models—models whose internal parameters are publicly available. Among the implications of open-source AI are: (1) Anyone with the requisite hardware can generate a working copy of the model and operate it independently—free from the monitoring that’s possible with a proprietary model. (Both OpenAI and Anthropic have used their control of the Application Programming Interface, or API, to spot and shut down users they say were using their proprietary models to conduct cyber intrusions.) (2) Anyone with the requisite hardware can amend the model, changing its parameters through customized training—possibly in ways that undermine not only safety “guardrails” that had been added to the model but deeper tendencies toward responsible behavior that had been trained into the model.
All of this helps explain why Nobel laureate Geoffrey Hinton, at least as responsible as any other single researcher for the “deep learning” revolution that paved the way for large language models, has said, “Open sourcing big models is like being able to buy nuclear weapons at RadioShack. . . Bad actors can fine tune them for all sorts of bad things.”
Open source advocates have their own arguments. Open source models, because they so readily elude centralized control, could in theory help prevent an unhealthy concentration of AI power—a danger that, in light of this technology’s capacity for surveillance and persuasion, is already considerable and will grow as models get more powerful. There are also purely economic arguments in favor of open source AI. Still, it seems likely that, as more and more powerful open source models emerge, there will be growing support, in various national capitols, for some kind of constraining policy. Even China, whose emphasis on open source models has in some ways served its interests, will find things to fear about a world with wholly unhindered open source development and proliferation.
Assuming policy constraints on open source AI indeed materialize, what form will they take? It’s hard to envision them clearly, but two assumptions seem sound:
(1) Any restrictive policies that the US or China or any other nation favors will be of limited value if they apply only at the national level. Getting a long series of numbers—a model’s parameters—across a nation’s borders is not a big challenge, even when the number of numbers exceeds a trillion. And once those borders have been breached, the model can be copied over and over, proliferating far and wide. Besides, even if you manage to keep a powerful open source model from thus entering your country, it can still, like powerful AI models in general, damage your country from afar—if, for example, it helps bad actors start a pandemic or stage an international cyberattack.
(2) Any policy that gives nations much reassurance will have to offer them not just transparency beyond their borders but an unusually fine-grained form of transparency. As AI chips grow in power and drop in cost, and such techniques as “distillation” create more efficient AI models, more and more formidable AI capabilities will assume a lower and lower profile. Right now there are consumer-level computers that cost less than $3,000 and can run—in the privacy of your home, with no internet connection required—a recently released and highly regarded Chinese open source model known as Qwen 3.8 27B. The model is no Mythos, but it’s no slouch—and next year the frontiers of DIY AI will expand further. And so too the following year…
Of course, the advance of AI could slow down significantly, or even stop, if some technical barrier to progress is reached. But lately the rate of advance has been speeding up. And both OpenAI and Anthropic are reported to already have the next generation of models in house, helping their workers develop subsequent generations. So it seems close to certain that America’s current approach to regulating AI—which consists basically of not regulating it, and engaging in only modest international dialogue about it—will soon be incompatible with public safety, let alone the public interest in a broader sense. And it seems likely that we will before long need not just robust international dialogue but meaningful international governance and indeed unprecedented kinds of international governance, complete with unprecedented degrees of international transparency.
In fact, I think the requisite transparency will be so fine-grained that we’ll need to cultivate what I call organic transparency—the kind of transparency that grows naturally out of deep economic, cultural, and scientific engagement among nations, as information flows via regular cross-border contact among elites. When two businesspeople from different nations sit down over drinks, you never know what new information will emerge. And when American and Chinese researchers chat at an international AI conference, they’ll learn more than they could have learned by just reading each other’s papers.
Of course, having this degree of engagement across all international borders would be unprecedented. It would require, among other things, a world without wars. Then again, artificial intelligence is unprecedented in the nature and magnitude of its powers. Nobody said managing the most abruptly momentous technological revolution in the history of the planet would be easy.
It’s tempting to dismiss this global adaptation to new security challenges—this revolution in the nature of international relations—as impossible, as irreconcilable with tribalistic human psychology and time-honored patterns of human behavior. But the big stories in AI safety of the past few months—the Mythos and Hugging Face stories—have had an appreciable impact on the psychology of important actors and large swaths of the public. There will be many more big stories to come out of AI, and they could have a galvanizing effect on humankind more broadly and more deeply.
That, at least, is the hope (along with the hope that these stories can be powerfully galvanizing without involving massive lethality). Certainly some powerful psychological jolts will be necessary. After all, the mission that technological evolution has assigned us is to finally do what humankind tried but failed to do in the aftermath of the twentieth century’s two biggest catastrophes, World War I and World War II: cross the threshold to true global community, a community cohesive enough to act with unified purpose. Crossing this threshold would mean that our species had, in a non-trivial sense, become more enlightened. It would also mean we’d realized part of John von Neumann’s original conception of the singularity: Human affairs as we’ve known them would not continue.
P.S.: Again, all five parts of this essay can be found in unified form here. And if a mere 7,500 words isn’t enough of a challenge, you can always check out my book The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning. The introductory chapter and excerpts from all other chapters are here.
Banners and graphics by Clark McGillis.