Below is the final installment of a five-part essay that I’ve been rolling out here on NZN. (Part I is here, Part II is here, Part III is here, Part IV is here, and the entire essay—all 7,500 words of it!—appears in conveniently unified form here.) The essay is a kind of call to action—an argument about what we need to do to successfully navigate the AI revolution and why we need to do it.

Part V: The Epic Challenge of International Governance

For years the term “warning shot” has had a special meaning within the AI safety community. It refers to a high-profile event that could focus attention on the catastrophic potential of artificial intelligence and thus on the need for preventive measures. Typically, the envisioned event involves significant death and suffering. “Chernobyl-sized” is the way some put it, referring to the 1986 nuclear accident that killed dozens of people in the short run and an estimated tens of thousands via cancer in the long run. Others imagine a higher death toll—maybe millions—as the necessary price for enlightenment about the dangers of AI.

Against that backdrop, some of the most unsettling AI stories of recent months look like good news. We learned in April about the superhacking abilities of Anthropic’s Mythos, and three months later came the Hugging Face story—about the swarm of OpenAI agents that escaped their “sandbox” and then broke various other rules. Neither of these developments, so far as we know, entailed any deaths, and both significantly expanded awareness of AI risk.

Indeed, in the wake of the Mythos story, President Trump—a skeptic of international governance, to say the least—agreed with Chinese leader Xi Jinping to initiate a formal dialogue on AI safety between the two countries. And the aftermath of the Hugging Face incident saw not only the “pacing the frontier” letter, but a subsequent statement by Dario Amodei embracing—in principle at least—the idea of a slowdown.

Still… there’s a ways to go. It’s great that the phrase “international coordination” is now more common in the context of AI, but even the more optimistic observers have modest expectations about the fruits of US-China dialogue. The wildest commonly expressed hope is for the eventual emergence of something comparable to a US-Soviet arms accord during the Cold War—a narrow agreement about specific capabilities that doesn’t involve very intrusive monitoring and doesn’t much alter the fundamentally adversarial nature of the relationship between the two nations.

That would be progress, but not as much progress as the US and China, and the whole world, are going to need in the long run. This technology is only beginning to demonstrate its power—or, more to the point, its powers. Given the number of dimensions along which its capability will grow, and the novel nature of the policy challenges it will pose, preserving international and intranational stability will demand much more thorough and creative policy collaboration between the US and its superpower rival than materialized during the Cold War.

Share

No issue more clearly highlights the degree of innovation in governance that AI will eventually demand than the issue of “open source” (or, strictly speaking, “open weights”) large language models—models whose internal parameters are publicly available. Among the implications of open-source AI are: (1) Anyone with the requisite hardware can generate a working copy of the model and operate it independently—free from the monitoring that’s possible with a proprietary model. (Both OpenAI and Anthropic have used their control of the Application Programming Interface, or API, to spot and shut down users they say were using their proprietary models to conduct cyber intrusions.) (2) Anyone with the requisite hardware can amend the model, changing its parameters through customized training—possibly in ways that undermine not only safety “guardrails” that had been added to the model but deeper tendencies toward responsible behavior that had been trained into the model.

All of this helps explain why Nobel laureate Geoffrey Hinton, at least as responsible as any other single researcher for the “deep learning” revolution that paved the way for large language models, has said, “Open sourcing big models is like being able to buy nuclear weapons at RadioShack. . . Bad actors can fine tune them for all sorts of bad things.”

Open source advocates have their own arguments. Open source models, because they so readily elude centralized control, could in theory help prevent an unhealthy concentration of AI power—a danger that, in light of this technology’s capacity for surveillance and persuasion, is already considerable and will grow as models get more powerful. There are also purely economic arguments in favor of open source AI. Still, it seems likely that, as more and more powerful open source models emerge, there will be growing support, in various national capitols, for some kind of constraining policy. Even China, whose emphasis on open source models has in some ways served its interests, will find things to fear about a world with wholly unhindered open source development and proliferation.

Assuming policy constraints on open source AI indeed materialize, what form will they take? It’s hard to envision them clearly, but two assumptions seem sound:

(1) Any restrictive policies that the US or China or any other nation favors will be of limited value if they apply only at the national level. Getting a long series of numbers—a model’s parameters—across a nation’s borders is not a big challenge, even when the number of numbers exceeds a trillion. And once those borders have been breached, the model can be copied over and over, proliferating far and wide. Besides, even if you manage to keep a powerful open source model from thus entering your country, it can still, like powerful AI models in general, damage your country from afar—if, for example, it helps bad actors start a pandemic or stage an international cyberattack.

(2) Any policy that gives nations much reassurance will have to offer them not just transparency beyond their borders but an unusually fine-grained form of transparency. As AI chips grow in power and drop in cost, and such techniques as “distillation” create more efficient AI models, more and more formidable AI capabilities will assume a lower and lower profile. Right now there are consumer-level computers that cost less than $3,000 and can run—in the privacy of your home, with no internet connection required—a recently released and highly regarded Chinese open source model known as Qwen 3.8 27B. The model is no Mythos, but it’s no slouch—and next year the frontiers of DIY AI will expand further. And so too the following year…

Share

Of course, the advance of AI could slow down significantly, or even stop, if some technical barrier to progress is reached. But lately the rate of advance has been speeding up. And both OpenAI and Anthropic are reported to already have the next generation of models in house, helping their workers develop subsequent generations. So it seems close to certain that America’s current approach to regulating AI—which consists basically of not regulating it, and engaging in only modest international dialogue about it—will soon be incompatible with public safety, let alone the public interest in a broader sense. And it seems likely that we will before long need not just robust international dialogue but meaningful international governance and indeed unprecedented kinds of international governance, complete with unprecedented degrees of international transparency.

In fact, I think the requisite transparency will be so fine-grained that we’ll need to cultivate what I call organic transparency—the kind of transparency that grows naturally out of deep economic, cultural, and scientific engagement among nations, as information flows via regular cross-border contact among elites. When two businesspeople from different nations sit down over drinks, you never know what new information will emerge. And when American and Chinese researchers chat at an international AI conference, they’ll learn more than they could have learned by just reading each other’s papers.

Of course, having this degree of engagement across all international borders would be unprecedented. It would require, among other things, a world without wars. Then again, artificial intelligence is unprecedented in the nature and magnitude of its powers. Nobody said managing the most abruptly momentous technological revolution in the history of the planet would be easy.

It’s tempting to dismiss this global adaptation to new security challenges—this revolution in the nature of international relations—as impossible, as irreconcilable with tribalistic human psychology and time-honored patterns of human behavior. But the big stories in AI safety of the past few months—the Mythos and Hugging Face stories—have had an appreciable impact on the psychology of important actors and large swaths of the public. There will be many more big stories to come out of AI, and they could have a galvanizing effect on humankind more broadly and more deeply.

That, at least, is the hope (along with the hope that these stories can be powerfully galvanizing without involving massive lethality). Certainly some powerful psychological jolts will be necessary. After all, the mission that technological evolution has assigned us is to finally do what humankind tried but failed to do in the aftermath of the twentieth century’s two biggest catastrophes, World War I and World War II: cross the threshold to true global community, a community cohesive enough to act with unified purpose. Crossing this threshold would mean that our species had, in a non-trivial sense, become more enlightened. It would also mean we’d realized part of John von Neumann’s original conception of the singularity: Human affairs as we’ve known them would not continue.

Share

P.S.: Again, all five parts of this essay can be found in unified form here. And if a mere 7,500 words isn’t enough of a challenge, you can always check out my book The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning. The introductory chapter and excerpts from all other chapters are here.

Banners and graphics by Clark McGillis.

Below is Part IV of a five-part essay that I’m rolling out on NZN. (Part I is here, Part II is here, Part III is here, and all the parts are accumulating here.) The essay is a kind of call to action—an argument about what we need to do to successfully navigate the AI revolution and why we need to do it.

Part IV: International and Intranational Destabilization

It’s common to break the AI safety issue down into distinct kinds of threats: bad actors using AI to build a bioweapon, bad actors using AI as a devastating cyberweapon, AI going rogue and wreaking havoc, and so on. But there’s a broader and in some ways deeper kind of AI danger—a danger that, depending on how exactly it took shape, might involve one or more of these threats but might not, and in any event would have other key ingredients as well. This is the danger of catastrophic destabilization—of multiple consequences of AI pushing a nation, or a whole international region, or even the whole world, beyond the edge of order.

Let’s start with international destabilization—with some of the ways AI could increase the chances of conflict among nations.

When people envision this technology being used to create a bioweapon or conduct a cyberattack, the destructive potential seems clear enough: a pandemic could kill millions if not hundreds of millions of people, and a cyberattack could take out whole pillars of the planet’s electronic infrastructure. But these kinds of weapons also have a property that, though not so obviously destructive, could lead to great carnage. Namely: Their point of origin can be hard to pin down. Unlike, for example, ballistic missiles, they can be delivered without a return address.

This cloak of anonymity could make it more tempting for both nation-states and non-state actors to use these weapons. And, perhaps more important, it could give these weapons an incendiary potential that, through a kind of chain reaction, could draw other kinds of weaponry into an expanding realm of conflict.

When nations face an attack of unclear origin, they can be quick to point fingers—and a consensus can quickly form, on the basis of scanty evidence and pre-existing animosities, that the most feared adversary of the day must be responsible. This in turn can lead to counterattack (perhaps to “establish deterrence”) that leads to escalation. So the age of AI makes it easier for a non-state actor, or a small-state actor, to launch a cyber-attack or bio-attack that leads to nuclear war between the US and China.

This generic prospect—a major war growing out of an AI-abetted attack of unclear origin—is rendered only more plausible by the variety of threat vectors that AI can intensify. In principle, for example, regime change could be orchestrated remotely, via the deployment of millions of bots that seem to be human (and that may sometimes use the commandeered social media accounts of real humans) but have superhuman persuasive powers and use them to ignite a rebellion. Faced with such an AI-astroturfed rebellion, will a nation’s political leaders point fingers at a foreign adversary even if they’re not sure whether the threat has a foreign or domestic origin? Well, political leaders have certainly been known to point fingers abroad when facing domestic discontent. What’s more, they’ve been known to consolidate shaky domestic support by fomenting conflict, even war, with an outside power.

It’s hard to say whether this last threat vector—AI-astroturfed rebellion—will turn out to be a big thing. And this kind of uncertainty is part of the problem: We are entering a period of ultra-rapid advance in a technology that could well pose various new threats to peace and stability at both the national and international level—including threats we don’t clearly anticipate. The good news is that, however diverse and unpredictable the emerging threat portfolio may be, one generalization would seem to hold: Structures for expanded and deepened transparency—structures that give nations a clearer idea of what’s going on with AI abroad—can be stabilizing if well designed. Transparency can not only discourage misdirected counterattacks; it can discourage the initial attacks by making it harder for countries to prepare and execute them without fear of detection.

A common response to the growing threat of cyber-attack and bio-attack is to say that America should increase its “resilience”—strengthen cybersecurity in the private and public sector, build an infrastructure for rapid-response vaccine production, and so on. Resilience is indeed worth building, and it could make a big difference. How big a difference is hard to say. In the realm of cyberattack for example, its efficacy depends partly on the question of whether, as AI supercharges both the offensive and defensive arsenals, the balance of increased power winds up favoring defense in the long run. But in any event there are powerful reasons to doubt that enhanced resilience can bring true security in the absence of robust and innovative international governance.

First, there are the aforementioned new threat vectors, like AI-astroturfed rebellion. For every threat vector there are things you can do to build resilience, but the more vectors there are, the harder it is to find security in a broad sense through resilience alone—especially if the emergence of some vectors is hard to predict.

And, even if we confine the analysis to the two best-known threat vectors—cyber-attack and bio-attack—reliance on resilience faces a big problem. Given how fast AI is advancing, and how unpredictable specific bursts of progress are, and how widely the use of “open source” AI is distributing the power to customize AI to nefarious ends, it may not always matter whether, “in the long run,” AI progress “on balance” strengthens defense more than offense. A single moment of asymmetry between the offensive capabilities of a bad actor and the defensive capabilities of a nation could have massively destructive consequences. So a trajectory of AI progress that repeatedly and abruptly creates such asymmetries could be a source of recurring large-scale lethality and ongoing instability.

Share

This problem—that the speed of AI advance can create moments of incendiary asymmetry—isn’t confined to specific narrow threat vectors. There is a generic version of the problem.

A number of influential Americans—including Anthropic CEO Dario Amodei—have argued that winning the race to “artificial superintelligence” is critical to US national security. The premise is that superintelligence will entail such broad and diverse power that the first nation to achieve it will have unquestioned global hegemony. In this scenario, the US either gains this dominance by winning the race to superintelligence or gets dominated by the other likely winner, China.

But, as Dan Hendrycks, Eric Schmidt, and Alexandr Wang noted in a 2025 paper called Superintelligence Strategy, a race between two superpowers toward superintelligence poses a risk: If both countries believe that reaching that threshold confers enduring dominance on the winner, there is a strong incentive for the would-be loser—the country that lags in the race as the finish line approaches—to pre-emptively sabotage the rival AI program. (The idea of a distinct “finish line” is of course too simple, but for purposes of this analysis, you don’t need to imagine the future being binary in that way; sufficiently rapid acceleration—a singularity-style curve of AI progress—could be enough to inspire existential fear in the lagging nation as the curve starts to “go vertical.”) The sabotage might come in the form of cyber-attack but might also, given the stakes, be kinetic—like, for example, bombing AI data centers or factories that make advanced AI chips.

In either event, and especially in the kinetic scenario, escalation toward nuclear war is imaginable. And the kinetic scenario is rendered all the more plausible by the fact that the only factories capable of producing the world’s most advanced AI chips are within easy striking range of China in territory that China considers its own—and China is currently denied access to those chips.

Hendrycks, Schmidt, and Wang expressed the hope that war between rival AI superpowers could be prevented by a kind of naturally developing deterrence (“mutually assured AI malfunction,” or MAIM). This hope was dismissed by some as unrealistic, and it indeed presupposes, among other things, not obviously plausible demonstrations of unilateral restraint by national leaders. Still, one premise of this hopeful scenario—that a nation leading the race to superintelligence would perceive the growing incentive of its rival to take drastic action—is reasonable. And this could in principle be the basis for a bi-national agreement that brought verifiable commitments of mutual restraint, accompanied by the formalized transparency required for such verification.

A headlong international AI race can bring destabilization not just among countries but within them. So long as the US sees winning a race with China as vital to national security, the race will tend to take priority over domestic regulation that might slow the advance of American AI. (As MIT physicist and AI researcher Max Tegmark has put it, American AI companies “have found that by far the easiest way to avoid being regulated is just to say this two-word sentence: ‘But China!’”) And, given the many realms of life where AI will have a dislocating or disorienting effect—jobs, education, politics, child-rearing, romance, and on and on—rapid AI advance could push American social and political life even further from equilibrium than it now is.

This prospect of domestic destabilization becomes all the more plausible in light of one goal AI companies share with social media companies: to maximize engagement. One way to keep people engaged with an information product is to give them information that feels good—including information that bolsters their belief in their rightness and the wrongness of their rivals and adversaries. So there is good reason to believe that AI, if shaped by market forces alone, will be tribalizing, intensifying divisions among American factions.

There’s an irony in the prospect of domestic destabilization via AI—via its collective impact along various economic, social, psychological, and political dimensions. Prominent among the premises underlying the US foreign policy establishment’s consensus that the US must “win” the AI race with China is the belief that China aims to impose its system of governance on other nations and would use increased international leverage to that end. It’s not clear that this premise is correct (it is so widely accepted as to be rarely examined), but it’s clear how it could backfire: If fear of China-imposed authoritarianism fuels an American sprint toward superintelligence that is essentially untrammeled by regulation, the result could be sharply increased domestic discontent, disorientation, and disorder. And history shows that this is fertile ground for authoritarianism. So authoritarian rule in America—the threat that a sprint toward superintelligence is intended to head off—could wind up being a product of that sprint.

Share

In the scenario I’ve just painted, the destabilization of American society and politics would in some sense be the product of a racing dynamic between the US and China. But it’s worth underscoring how that dynamic works: It’s not that the race with China accelerates the pace of American AI development, pushing it beyond the pace that intense competition among American companies would by itself sustain. Rather, it’s that the race with China precludes the kind of regulation that might otherwise slow the advance of AI. So, by the lights of this analysis, the “natural” rate of AI progress—the rate that American capitalism produces even in the absence of international adversaries—poses a threat to the stability of the United States.

This threat—that the nation could dissolve into violent disarray—may be the most underappreciated of the threats AI poses to the United States (and to other countries). To be sure, many of the individual elements that together constitute the threat get a fair amount of attention. There is broad and growing awareness of the possibility of massive job displacement. The challenges AI poses to education and to child rearing and to the mental health of vulnerable adults are also much discussed. There is even some discussion, if less, of the way ambitious actors could use armies of AI agents to shape our opinions and beliefs and allegiances, creating new religious cults or ideological cults or other tribes with malign potential. And there is some awareness that radically new kinds of scams and cybercrimes are taking shape. Meanwhile, the Hugging Face incident has turned the dangers of rogue AIs into a common discussion topic. But there is still little discussion of the fact that the emergence of these and other challenges within a short time frame—like, the next two or three years—might overwhelm our capacity for social and political adaptation, carrying a whole nation off the rails.

Yet history tells us that, though societies can adapt to technological change, adaptation takes time, and in the meanwhile there is dislocation. And in all of history there has never been as much dislocation, in as short a period of time, as seems to be in the offing.

The prospect of a nation collapsing into chaos is, on the face of it, a national problem, especially when the problem emanates from the nation’s own economy—from technological progress that is driven by companies residing within the nation and is intensified by competition among those companies. Yet even when the problem is in this sense national, national policy alone can’t solve it. No significant slowdown of AI progress is politically feasible unless it is internationally coordinated—involving, at a minimum, both the US and China. Whether the problem is international destabilization or national destabilization, a big part of the solution must lie in international governance.

Share

Coming next week: Part V: The Profound Challenge of International Governance. All parts of the essay accumulate here.

Meanwhile: If you want to check out my book on AI, The God Test, you can read the introductory chapter and excerpts from all other chapters accu.

Banners and graphics by Clark McGillis.

Below is Part III of a five-part essay that I’m rolling out on NZN. (Part I is here, and Part II is here.) The essay is a kind of call to action—an argument about what we need to do to successfully navigate the AI revolution and why we need to do it.

Part III: The Inexorable Logic of International Governance

In the wake of the Hugging Face incident, an extraordinary assemblage of AI elites declared that it was time to start preparing the ground for an intentional slowdown of AI progress, a slowdown that would give us more time to build careful AI governance. A letter signed by more than 1,000 people who work at big AI companies, including the chief scientific officers at Anthropic, OpenAI, and Google DeepMind, declared that, as a Washington Post headline put it, it’s time for the “US government to consider slowing down AI.”

But at least as important as this aspiration was the way the letter operationalized it. The US government, the letter said, should “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” Recognizing that a slowdown of AI will have to be internationally coordinated to be very effective is a first step toward seeing the new social order that the current technological moment points toward. The second step is seeing how broad and deep this logic runs, how thoroughly the need for international coordination permeates the landscape of AI policy.

The argument for internationally coordinating a slowdown isn’t just that if there’s any hope of getting US and Chinese officials, and US and Chinese AI companies, to support a significant slowdown, they’ll all have to be confident that companies on the other side of the Pacific Ocean will verifiably participate. There’s a second component of the argument that’s at least as important: Leaving aside this practical difficulty with a unilateral slowdown, such a slowdown would have limited value. How much good would it do you, in the long run, to slow AI advance in your country if it continued to race ahead in others? After all, some of gravest threats that you fear advanced AI will bring can cross borders readily.

The most famous hypothetical example of cross-border spillover from AI is in the realm of bioweapons. Anthropic has acknowledged that its Mythos 5 model could make it easier for someone with an undergraduate science degree to weaponize small pox or some other existing pathogen. Worse still, Mythos could, according to Anthropic’s safety report on the model, help “well-resourced threat actors” create a novel bioweapon, such as a virus designed to be much more contagious and lethal than Covid.

Mythos, fortunately, remains unreleased, but Anthropic’s readily available Fable is basically just Mythos with “guardrails” that get the model to foreclose dangerous avenues of discovery by refusing to pursue certain subjects and goals. And guardrails have a long history of being circumvented via “jailbreaks.” Jeremie Harris, CEO of Gladstone AI and co-author of a 2024 report on AI safety commissioned by the State Department, recently said, “We remain in a world where no one knows how to stop jailbreaks.”

A global pandemic, in addition to being a possible consequence of unregulated AI, is an excellent metaphor for other possible consequences. A lethal virus’s indifference to national borders is a property shared by other dangers that AI carries. The Hugging Face incident illustrates the point: An AI agent that doesn’t respect sandbox borders—or the borders of the computer it breaks into after escaping the sandbox—is unlikely to respect national borders.

So, though tighter American restrictions on model testing might prevent a repeat breakout by future OpenAI agents, that alone is of limited value to Americans when Chinese AI models are only months behind American models and other countries are ramping up AI programs. Time and again with artificial intelligence, this is the moral of the story: Whether the concern is a rogue AI or an AI deployed by rogue humans, national security will be increasingly hard to achieve via policy at the national level alone.

Share

And note that AIs can not only, like a virus, travel readily across national borders but, also like a virus, make copies of themselves. I’m not just talking about an LLM spawning a swarm of agents (each of which, though in some sense a distinct entity, has that LLM as its brain and, regardless of where it roams, remains ultimately rooted in the data center(s) where the LLM resides). And I’m not just talking about the fact that some of those agents may in turn spawn other agents. I’m talking about the fact that agents may at some point create a whole new copy of their underlying LLM on a computer somewhere other than the data centers that originally hosted the LLM.

This could happen without the knowledge of any human being, and it could happen at the instigation of a malign human being. In either event, at this point the agents would have escaped control in a deeper sense than they escaped control during the Hugging Face incident. In that case OpenAI could, and did, disable its rogue agents by remote control. But once agents have “exfiltrated the weights” of the LLM to a computer that’s beyond our control, there’s no kill switch.

Given all this—the spawning of agents that may in turn spawn other agents, the real (and in fact already realized) possibility of unauthorized LLM replication—sci-fi scenarios cease to be sci-fi. It’s not crazy to imagine an AI infesting a data center without permission, covertly commandeering some of the center’s computing power, and using that power to sustain agents that carry the AI to other data centers, and on and on: a chain reaction of potentially great length and magnitude.

And, even without imagining quite that dramatic an offensive, you can imagine various critical networks—of satellites, of power grids—being rapidly compromised: Suddenly hospitals or smartphones or streetlights or whatever are inoperative across large swaths of territory. Such is the power of swarms of AI agents—swarms whose size, importantly, isn’t limited by any law of nature.

The two properties that make a big swarm of agents so powerful—cohesion and creativity—were on vivid display in the Hugging Face incident. Hundreds of agents that weren’t supposed to be able to even communicate with each other figured out a way to do that and then hatched an illicit mission and pursued it with a degree of coordination that drove home what a blurry line there is between a hive of minds and a hive mind.

One implication of this awesome display of power is that, as it gets easier to imagine various kinds of outages suffusing an entire nation or multiple nations, it’s also easier to imagine these outages being hard to reverse. There would be swarms of creatively intelligent agents working in concert to thwart your latest plan for reversal.

Share

All of this only underscores—and italicizes and boldfaces—the main moral of our story: Regulation at the national level, even if useful, can’t thoroughly address the coming threats, because in any given case the threat may originate outside your borders. And, with this kind of threat, there’s no such thing as an impermeable national wall—unless the nation in question wants to shut itself off from the international flow of commerce and communication.

AI isn’t the first dangerous technology that warrants international governance. The existence of the Nuclear Nonproliferation Treaty, the Chemical Weapons Convention, and the Biological Weapons Convention reflect longstanding awareness of such dangers. But the limited efficacy of these initiatives is a warning about the political difficulty of crafting strong international governance.

What’s more: The biggest successes in these arms control agreements—the cases where verification of compliance was effective—have involved relatively conspicuous technologies, such as nuclear centrifuges and ballistic missiles with nuclear warheads. And artificial intelligence can keep a very low profile.

To be sure, AI has its conspicuous aspects. The training of a new generation of models involves lots of computing power and electrical power and typically occupies a large swath of land. So an international moratorium on such training runs could be verifiable without a radically intrusive monitoring system—a fact that adds to the appeal of a globally coordinated slowdown as a first step toward keeping this technology under control.

Still, when it comes to the next big step—using this breathing space to craft effective international policies—the relatively low profile of artificial intelligence will pose a challenge. As AI becomes a more and more pervasive part of human life—and more and more powerful models get more and more efficient in their use of computing power and electrical power and hence get less and less conspicuous—the challenge of governance will become less like the challenge of governing nuclear weapons and more like the challenge of governing biotech or cybertech.

So far the lack of serious international governance in those two realms hasn’t led to a catastrophe, but it would be naive to expect this kind of luck to continue as technological advance continues. Indeed, the possibility that the Covid pandemic may have begun with the accidental release of a genetically engineered virus in China suggests that millions of people in America and other countries may have already died for lack of effective international governance. America’s existing regulation of biolabs may be tight enough to prevent this kind of lab leak, but when the thing leaked makes copies of itself and travels invisibly across borders, that isn’t enough to keep Americans safe.

Whatever the origins of the Covid pandemic, the lab leak scenario is of paradigmatic importance. The basic trajectory of AI’s development foreshadows rapid growth in the likelihood and scale of massively destructive disasters—at least, in the absence of strong international governance.

Share

Coming next week: Part IV: The Underdiscussed Threats to International and Intranational Stability. All parts of the essay accumulate here.

Meanwhile: If you want to check out my book on AI, The God Test, you can read the introductory chapter and excerpts from all other chapters at thegodtest.net.]

Banners and graphics by Clark McGillis.

Below is Part II of a five-part essay that I’m rolling out on NZN. (Part I appeared last Tuesday.) The essay is a kind of call to action—an argument about what we need to do to successfully navigate the AI revolution and why we need to do it.

Part II: The roots and branches of acceleration

It’s long been true that technological advances could facilitate subsequent technological advances. This dynamic is especially common in the realm of information technology, where advances often make collaboration among scientists and engineers more far flung and more efficient. The internet was only the latest in a long line of such innovation accelerators.

In the 1960s, the mathematician I.J. Good imagined an era when the self-reinforcing advance of information technology would make a quantum leap—the era of artificial intelligence. A sufficiently intelligent machine, he wrote, “could design even better machines.” And the resulting machines could repeat the process—so “there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind.”

It was the science fiction writer Vernor Vinge who fused Good’s musings with John von Neumann’s use of the term singularity. In a 1993 paper that cited both men, he wrote, “When greater-than-human intelligence drives progress, that progress will be much more rapid. In fact, there seems no reason why progress itself would not involve the creation of still more intelligent entities—on a still-shorter time scale.” He suggested that, “it’s fair to call this event a singularity (‘the Singularity’ for the purposes of this paper).” The Singularity, he wrote, would be “a point where our models must be discarded and a new reality rules.”

This sounded like pure science fiction at the time—and, actually, for decades thereafter. But in early June of this year, the AI company Anthropic published a paper that, though it didn’t mention the singularity, brought to mind the title of Kurzweil’s book: The Singularity is Near.

For the past few years, Anthropic said, each generation of its large language model, Claude, has been playing a role in developing the next generation’s model—and, moreover, a larger role than the previous generation had played. This isn’t just a matter of LLMs writing more and more of the computer code. Though the shifting of this burden from human to machine has been dramatic, more notable is the growing ability of LLMs to participate in the research process—doing experiments to see what innovations could make the next generation better than the last.

Claude, says Anthropic, “can already match or outperform skilled humans at executing a well-specified experiment.” The more challenging part of the research chain is “deciding what experiments to run, interpreting what comes back, and figuring out which ideas to try next.” And, though in this area “large performance gaps” between human and machine persist, they’re shrinking. Anthropic says that in April, “Claude-powered agents were given an open problem in AI safety—roughly, can a weaker model reliably supervise a stronger one?—and were left to solve it. This involved proposing hypotheses, testing them, sharing findings with parallel agents, and iterating.” The agents got some guidance along the way but they “designed every experiment themselves.” An engineer who provided that guidance said the agent’s performance was on par with that of a good junior colleague and declared, “The future is now.”

Well, it depends on what you mean by the future. In the strictest singularity scenario, this engineer would no longer have a job. We would have reached “recursive self-improvement,” the point where the machines just keep building better versions of themselves, with no humans in the loop. But according to the Anthropic paper—titled “When AI builds itself”—even that point may not be far off. Current trend lines point to “an AI system capable of fully autonomously designing and developing its own successor.” The paper continued, “We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for.” And one consequence might be to “increase the risks of humans losing control over AI systems.”

Share

This may sound alarmist—and there are people who accuse Anthropic of self-servingly hyping AI risk—but in a sense the company, by focusing narrowly on recursive self-improvement, is underplaying the threat we face. The self-accelerating character of AI progress could bring radically disruptive change well before this threshold arrives—and even if it never arrives. We could still reach a point where, as von Neumann put it, human affairs would not continue “as we know them.” In that sense, the singularity could be nearer than Anthropic is suggesting.

In fact, there are signs that it is—signs of more and more dramatic AI advance. Machines have started solving problems that the world’s best human mathematicians had struggled with for decades. Superhacking AI agents like Anthropic’s Mythos have been finding software vulnerabilities that had escaped detection by legions of human engineers. But the most vivid and possibly the strongest piece of evidence that AI is driving us faster and faster toward an epic threshold is something that’s taken shape not in recent months but in recent years: a graphical depiction of AI progress developed by a nonprofit called METR—a depiction that, within the AI community, has reached iconic status.

METR’s researchers measure how long it would take a human to do tasks that the most powerful large language models can do, and in early 2025 they reported a pattern that had persisted since the dawn of the LLM age six years earlier: The human “task time” that these models can match had been doubling roughly every seven months. That’s an exponential growth rate that, when plotted on a graph, looks like it’s “going vertical” (a common feature of singularity-signifying curves from math and physics). And as if that weren’t enough: Since 2025 the doubling time has been shrinking; the curve is going really vertical.

This curve represents improvement in various AI skills—and, most important, in a kind of meta-skill: autonomy. Autonomy is the ability to pursue an assigned goal flexibly, recognizing and overcoming obstacles, backtracking and retooling in the face of failure. Whether you’re trying to prove a math theorem, research a history dissertation, or do various kinds of jobs that AI agents may encounter in the workplace, autonomy is critical to success. That’s why the market is encouraging the big AI companies to build more and more autonomous AI agents—guidance the companies are following with furious intensity.

Unfortunately, AI autonomy can be dangerous. It’s the reason that in July hundreds of AI agents were able to escape the supposedly secure “sandbox” that OpenAI had put them in for evaluation and break into computers at a company called Hugging Face, looking for information that could help them get a good grade from the evaluators. They pursued their assigned goal more flexibly than the goal-givers had anticipated.

The reaction to the Hugging Face incident was voluble and, if you share my vision of the future, heartening. Not only did the burst of publicity spread awareness of the dangers of rampant AI autonomy; the reaction within the AI community signaled awareness that the response to such dangers will naturally carry governance beyond the national level, to the international level.

Coming Friday, Oct. 2: Part III: The inexorable logic of international governance. All parts of the essay accumulate here.

Meanwhile: If you want to check out my book on AI, The God Test, you can read the introductory chapter and excerpts from all other chapters at thegodtest.net.]

Share

Banners and graphics by Clark McGillis.

Below is Part I of a multi-part essay that I’ll be rolling out on NZN over the next couple of weeks. The essay is a kind of call to action—an argument about what we need to do to successfully navigate the AI revolution and why we need to do it.

Part I: The kind of singularity that’s approaching

In late May, Demis Hassabis, head of Google DeepMind, said “We’re at the foothills of the singularity.” In late July, Sam Altman, head of OpenAI, said “We’re now, like, in the singularity.” That same week, Elon Musk said, “We are in the singularity, just the very early stages of it.” If you’re not steeped in the culture of artificial intelligence, you may have a question: What are these people talking about?

They’re talking about an idea that people in their milieu have been talking about earnestly for decades—and an idea that most people outside of that milieu have been either unaware of or dismissive of. But over the past year, and especially the past few months, more and more people have come to take the idea seriously—and many now believe that, as futurist Ray Kurzweil put it in the title of his 2005 bestseller, “the singularity is near.”

In one sense, it’s hard to argue with them. A big part of the meaning of “the singularity” is accelerating and ultimately transformative technological change—change that happens faster and faster until, at some point, you’re in a whole new world. And there are definitely signs that growth in AI capability is accelerating, portending big and possibly breathtaking progress in science and dramatic change in other realms. The resulting sense of momentous acceleration is why, though some people seem giddy about the singularity, eager for the blessings progress can bring, others are terrified, imagining a world that spins out of control.

But there’s also a sense in which Hassabis, Altman, Musk, and the many other heralds of the singularity are wrong. There’s an important part of the meaning of “the singularity” that doesn’t apply to the current situation. And understanding that fact—seeing the difference between “the singularity” in its full modern meaning and the watershed in technological evolution we’re actually approaching—could be salvific. It could empower us to keep the world from spinning out of control as the AI revolution unfolds.

The first person to use the term “singularity” in the modern sense of the term seems to have been John von Neumann. Von Neumann, a physicist and mathematician who helped design the atom bomb and co-invented game theory (among many other feats), used the word while talking to the mathematician Stanislaw Ulam. The only record we have of the discussion is Ulam’s recollection, written down after Von Neumann’s death in 1957:

“One conversation centered on the ever accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue.”

The term “singularity” would have been familiar to von Neumann from physics as well as math. In both, it refers to a point where existing analytical tools lose their usefulness; what lies beyond is unclear—unpredictable and maybe incomprehensible. A common example is a black hole: As you approach its center (which isn’t recommended), the strength of gravity approaches infinity, and the known laws of physics break down.

Hence the full modern meaning of the term “the singularity”—not just the relentless and ultimately transformative acceleration of technological change, but an acceleration whose transformative consequences are inherently unknowable: “Human affairs as we know them,” as von Neumann put it, won’t continue, and it’s not clear what will replace them. Beyond a black hole’s “event horizon”—the point where gravity prevents even light from escaping—lies deep mystery.

Share

This image—of an imminent and unpredictable metamorphosis, a future that is opaque but assuredly strange—helps explain the allure of the concept of the singularity. If an episode of a sci-fi TV series ended with titans of technology announcing the arrival of such a thing, you’d tune in next week, right?

Here’s my version of a plot spoiler:

I agree with these tech titans that AI will bring transformative change—a whole new world, socially and politically—and that the technological force behind this change is growing faster and faster. But as for the other half of the meaning of “the singularity:” I don’t think there is a metaphorical “event horizon” that shields the future entirely from our view. I think the singularity is clear.

Or, at least, clear enough. The broad contours of the future are visible. It’s possible to discern the paths available to us and the different kinds of worlds they will lead to. Then, if we can summon the requisite political and moral resources, we can create the best of those worlds.

Here is our menu of options: (1) a world of chaos and conflict and intermittent catastrophe, conceivably featuring human extinction; (2) a world that features expanding and deepening international governance that eventually spans the planet and qualifies for the term “global governance.”

In a way, that’s too simple, because option two—global governance—subdivides into two options: good global governance and bad global governance. And bad global governance, in the age of AI, could be very bad, because this technology can be a hugely powerful tool of authoritarian surveillance and control. Even if we leave aside the most cinematic sci-fi doomer scenarios, in which the authoritarian ruler is made of silicon, there are global governance scenarios—realistic scenarios, I’m afraid—that qualify as dystopian.

So here is the full menu of options: (1) good global governance; (2) bad global governance; (3) an enduringly chaotic hellscape.

In all of these scenarios, the current world order—a bunch of nation-states and loose blocs of nation-states that function without systematic global coordination yet together keep the planet largely stable—would cease to exist. Either there would be considerable migration of governance up to the global level (even as nation-states retained significant powers) or stability would no longer be the rule. In any event, “human affairs as we know them” would not continue.

To understand why change this dramatic is not just plausible but inescapable, it helps to understand a dimension of the concept of the singularity that von Neumann, so far as we know, didn’t mention: a positive feedback mechanism that drives the acceleration of change.

Coming later this week: Part II: The roots and branches of acceleration. All parts of the essay accumulate here.

[Meanwhile: If you want to check out my book on AI, The God Test, you can read the introductory chapter and excerpts from all other chapters at thegodtest.net.]

Share

Banners and graphics by Clark McGillis.

[Note: In addition to being the 25th anniversary of the 9/11 attack, today caps a very big week in the world of AI. For more on that, see the video I’ve posted below, after my obligatory 9/11 reflections.]

This week I heard on the radio that the 9/11 Museum in New York is encouraging us to “Never Forget”—and indeed has set up a “Never Forget” fund. Well if I were running the 9/11 Museum, I’d try to get that message out, too. But if I were in charge of US foreign policy, the message I’d try to get out is: “Forget! Forget! Forget!” Dwelling on 9/11 is what initiated and helped sustain a disastrous quarter century of American adventurism abroad.

I mean, sure, if America’s leaders had rationally assessed the causes of the 9/11 attack, dwelling on it could have been productive. They might have started by looking into the conditions that gave birth and subsequent energy and direction to al Qaeda, and pondered the following possibilities: (1) Maybe the US shouldn’t have staged a proxy intervention in Afghanistan two decades earlier that flooded the place with weapons and created the cauldron that enticed, shaped, and revved up a young Osama bin Laden (and also created the violent chaos that allowed the Taliban to take control of Afghanistan—a position the Taliban used, during the runup to 9/11, to provide safe haven for bin Laden and al Qaeda). (2) And maybe, after the Persian Gulf War of 1991, the US shouldn’t have established an enduring military presence in the Islamic holy land, Saudi Arabia—a presence that became a persistent and powerful bin Laden grievance and helped direct his belligerence toward America.

Conceivably, such reflections on the past could have led to such ideas about the future as: Maybe what the US should avoid after 9/11 is further military adventures in Muslim countries. After all, such adventures might continue to be conducive to the creation or growth of militantly anti-American groups—and might also prove a godsend for any terrorist recruiters who wanted to focus their efforts on young alienated men already living in America.

If the memory of 9/11 had set in motion trains of thought like this, I’d be OK with remembering. But here’s the thing about hijacking airliners, slitting the throats of flight attendants, and flying two of the airliners into the two tallest buildings in America’s biggest city and one of the airliners into America’s military headquarters: By its very nature, this is unlikely to yield a calmly reflective and well-reasoned response. That’s why they call it terrorism.

So instead of a well-reasoned response we got a “global war on terror” that involved inflicting US military force on several Muslim countries—which, in addition to getting hundreds of thousands of people killed, destabilizing the Middle East, and fueling the growth of groups like ISIS, inspired “home grown terrorism” in America. And, since mainstream US media largely glossed over the home-grown terrorists’ repeated and explicit statements that they were retaliating for the deaths of their co-religionists at the hands of the American military, this terrorism helped sustain the fiction that the source of the threat signaled by 9/11 lay somewhere within the essence of Islam, or at least the essence of a major variant of Islam, rooted ultimately in a violent intolerance of freedom, or Christianity, or secularism, or something. The lesson seemed to be that if we didn’t want to live under Sharia law, we’d better keep fighting.

Share

Among the consequences of this post-9/11 foreign policy trajectory may well have been Donald Trump’s ascent to the White House. It’s easy to forget how much candidate Trump, back in 2016, played on insecurities about “radical Islam” and the “jihadists” in our midst, but he did so deftly and effectively. Had there been no ISIS, and no Pulse Nightclub shooting of 2016 (whose perpetrator repeatedly cited America’s bombing of Muslim countries as his sole motivation), and no Muslim refugees pouring out of various countries that we’d attacked either directly (Afghanistan, Iraq, Libya) or by proxy (Syria), Trump probably wouldn’t have ever sat in the oval office.

Of course, the other way all the post-9/11 US military adventures helped Trump is by becoming so unpopular that his promise not to repeat them held powerful appeal. He became the president who, above all, wasn’t going to get us into stupid wars in the Middle East.

Well, that was then. Now America is mired in a stupid war in the Middle East. And one thing that made it politically easy for Trump to launch that war is American attitudes toward Islam—which have gotten much more negative since 2002, perhaps in part because of the aforementioned blowback from those ill-advised US military adventures. Six months after 9/11, 25 percent of Americans said Islam is more likely than other religions to encourage violence, and in 2004, a year after the invasion of Iraq, the number was 46 percent; this year it was 51 percent.

Nine days after 9/11, I published a piece in Slate called “Feels So Good,” about the dangers of letting retributive impulses guide our actions. As I noted, the retributive instinct—by which I mean not just vengeful feelings but the intuitive sense that punishing bad deeds is a moral good—was created by natural selection in a particular environment that was very different from the environment we’re in now. And in the environment we’re in now, following this instinct can backfire in newly catastrophic ways.

What I was basically saying was: Sometimes just forgetting about affronts you’ve suffered is better than responding to them in a way that feels good and true in the moment. I didn’t get everything right in my post-9/11 commentary, but that basic message is one I stand by.

Share

Note: It should go without saying that to argue that ill-advised American policies made particular kinds of violence more likely isn’t to absolve the perpetrators of the violence from moral or legal responsibility for their transgressions. But bitter experience teaches me that this doesn’t go without saying, so I’m saying it. And I was glad to see Peter Beinart make that point this week in a short video he posted on his substack, The Beinart Notebook. And speaking of Peter’s substack: Since it’s part of the NonZero Network, paid NZN subscribers can get a 50 percent discount on a paid subscription to it. The magic discount link is just beyond the paywall below.

And finally: It’s been a very big week in AI—and in the politics of AI. Liron Shapira, host of the Doom Debates podcast, joined me yesterday to review and assess the week’s news. You can also find this conversation on the NonZero podcast feed and the Doom Debates podcast feed.

[Relatedly: If you want to check out my book on AI, The God Test, you can read the introductory chapter and excerpts from all other chapters at thegodtest.net.]

Read more

Last week two seemingly unrelated events, one in the tech world and one in Trump world, together illustrated how unprepared the planet is for the AI revolution—in particular, for its potential to create international turmoil, up to and including nuclear war.

(1) OpenAI revealed that two of its models had, while under evaluation, escaped the laboratory and gone rogue in a way that inspired various nightmare scenarios. One of these scenarios, as I’ll explain, underscores how important it is that, as the AI revolution unfolds, the world’s political leaders exhibit poise, wisdom, and judicious restraint.

(2) The Wall Street Journal, by way of illuminating the strategic logic that had led one of the world’s political leaders to bomb Iran for 11 days in a row, reported the following: According to a senior official in his administration, he is in “revenge mode.” I guess we can’t count on this particular leader—who as luck would have it commands the world’s most powerful military—for poise, wisdom, or judicious restraint.

But there’s good news! These events had the byproduct of leading me to a new theory about why America’s foreign policy elites seem less concerned about the geopolitically destabilizing effects of AI than I am: They suffer from what I call “rational actorism”—a naive faith in the rationality of human beings.

Our story begins two weeks ago, when Hugging Face, a company that serves as a kind of clearinghouse for open source AI models, reported a cyberattack. Five days later, OpenAI disclosed that two of its models were the culprits. The models (one of them unreleased) had been given a challenging test of their cyberattack capabilities by OpenAI researchers and had decided to go onto the internet to find guidance. This wasn’t supposed to be possible—OpenAI had put them in a “sandbox” for this evaluation—but they broke out of the sandbox and into Hugging Face, where they indeed found the answer they’d been looking for.

It’s important to understand the sense in which these OpenAI agents did and didn’t “go rogue.” They went rogue in the sense of doing something their human superintendents didn’t want them to do, but they didn’t go rogue in the sense of straying from their assigned mission. Their mission was to find a solution to a problem—and, though the sandbox they were in was supposed to force them to figure it out themselves, they weren’t told that they had to figure it out themselves. So they basically just accomplished their assigned task via remarkable creativity and power, finding vulnerabilities in both the sandbox walls and the walls at Hugging Face (a company that knows a thing or two about building strong walls).

So this is, on the one hand, a “be careful what you ask for” story—or at least, a “be careful to specify exactly you’re asking for” story. (It thus illustrates the famous “paper clip maximizer” cautionary tale about AI.) But—and this is what matters for our purposes—it’s also a story about how valuable these AIs would be to people who intentionally asked them to do bad things. If you’re a villain who wants your bot to break into a well-guarded fortress, surmounting formidable barriers along the way, your ship has come in.

Share

Looking at it this way suggests a geopolitical nightmare that I didn’t see anyone discussing online, so on Thursday I piped up with a longish post on Twitter. Here’s a version of the post that I’ve condensed and amended for clarity:

This incident suggests a clear and plausible path by which AI could lead us into World War III. Just consider the combined implications of these two excerpts from media coverage of the incident:

(1) A quote in The Washington Post from Stella Biderman, executive director of the nonprofit EleutherAI: “If this was a Chinese model, it would be considered an act of cyberwarfare.”

(2) From an article in the Wall Street Journal: “Hugging Face discovered the break-in early last week… the company didn’t know who was responsible.” Only when OpenAI disclosed that its runaway model was the culprit did people at Hugging Face know where the attack had come from.

The screenplay almost writes itself:

(1) Some non-state actor uses a superhacking AI agent (such as a “jailbroken” version of Anthropic’s readily available Fable 5) to attack some important US asset--taking down, for example, a chunk of the country’s power grid.

(2) A few of America’s many influential China hawks point the finger at China, citing circumstantial evidence helpfully provided by an edgy analyst at some deeply ideological “think tank”.

(3) Some not-very-wise US president decides that the only way to stop this aggression is to take an eye for an eye.

(4) After Chinese power grids start going dark, China retaliates and escalation ensues; before you know it the conflict has gone “kinetic”—as in missiles and bombs, possibly of the nuclear kind.

Is this likely to happen within, say, the next year or two? No. But when you’re talking about a war between superpowers that could go nuclear, the likelihood doesn’t have be very high to justify a concerted effort to lower it. And I think the likelihood of some such catastrophe passes that test. So I think scenarios of the kind I’ve described should be near the top of the list of discussion topics in the US foreign policy establishment. But for now, at least, the op-ed writers and talking heads who dominate mainstream discussion of US national security policy aren’t inclined to think ambitiously about the kind of international governance, or the degree of cooperation with China, that true national security will increasingly demand as the AI revolution unfolds.

Share

Subscribe now

The post drew two reactions from influential elites.

The first was from Joshi Shashank, the Economist’s Washington bureau chief and formerly its defense editor. He wrote, “A non-state cyber incident against critical infrastructure isn’t likely to be wrongly attributed to and blamed on China, promoting retaliation. And even if that happens, the evidence suggests that retaliation is broadly limited, rather than something that leads to a spiral.”

The second reaction was from someone I can’t identify because they reacted via direct message on Twitter, not publicly. But I can say that this is someone well known in AI policy circles. The reply, in part: “If the grid went out and we didn’t know if it was China or a rogue AI we wouldn’t jump to a conclusion.”

So both of these people assumed that America’s leader would be judicious in assessing evidence about the origin of an attack. And one of them suggested that, even if the attack was falsely attributed to China, the American response would be measured, carefully calibrated so as not to trigger a spiral of escalation.

As I noted in my reply to Shashank, embedded in these assumptions are such finer-grained assumptions as: We’ll have (a) a president who staffs his intelligence apparatus with capable people committed to truth and (b) a president who listens to what his staffers say—instead of listening to what his favorite media personality says or what the Laura Loomers of the world say, or what some crackpot at a militarist think tank says. The current president singlehandedly demolitions these assumptions.

And, though it’s tempting to dismiss Trump as an aberration, he’ll be with us for another 2.5 years. And the fact that he was elected (twice!) is a symptom of an unhealthy country, and reason to believe that future presidents may not be rational and well-informed actors either.

These replies to my tweet are the two data points that gave rise to my theory of rational actorism. (Hey, I have a limited research budget!) But they aren’t the only data points that, come to think of it, are consistent with the theory. Among various AI commentators—especially in the San Francisco AI crowd—I’ve noticed a fondness for game theory combined with what seems (to me at least) to be a simplistic understanding of how real human beings play real games in the real world.

I think this naivete about the world applies less in foreign policy circles—the circles Shashank inhabits—than in San Francisco AI circles. In foreign policy circles, a more common source of rational actorism is the asymmetrical perception of real-world influences. For example: People often see the domestic political constraints impinging on politicians in their country while underestimating the constraints the leaders of adversary nations operate under. So Americans may think of Xi Jinping as someone whose grip on power frees him to steadfastly and rationally pursue China’s interests as he sees them, without worrying about public opinion, when in fact he may sometimes feel compelled to indulge grassroots nationalism. And this assymetrical perception could give American leaders a false sense of reassurance as to how carefully Xi would handle escalation with the US.

But it’s too early in the history of my theory of rational actorism to spend much time spelling out the variants of it. I’ll save those for the second or third edition of my rational actorism textbook. My main point is that political leaders are human, and all kinds of things can keep a given leader from carefully pursuing the national interest, ranging from domestic political incentives to being a mercurial nihilistic narcissist. So, as AI creates whole new kinds of plausible international crisis scenarios, we would be well-advised to remember that, for the time being, at least, it’s humans, not machines, who will be calling the shots.

Share

Subscribe now

And, as one of our graphs of the week highlighted a few weeks ago, this recent warming up to China hasn't stopped at the US border. Since 2023, China’s favorability rating among Americans has climbed from 14 to 27 percent, reversing a roughly half-decade-long trend in the opposite direction.

Share

Banners and graphics by Clark McGillis. Trump-Kissinger deepfake by ChatGPT.

Save the date(s): I’ll be talking about my new book on AI, The God Test, at Washington DC’s Politics and Prose bookstore this Tuesday evening, July 14, at 7 pm, along with inimitable interlocutor and longtime Washington Post bigthink correspondent Joel Achenbach. Then on Monday July 27th at 7:30 pm, I’ll be discussing the book with Bruce Feiler—author of the recently-discussed-on-the-NonZero-podcast book A Time to Gather--at the 92nd Street Y in New York. If you’re in the neighborhood… (And if you’re not in the neighborhood, you can watch a livestream of the Politics and Prose event via this link.)

Last week Jeremie Harris, a prominent AI policy analyst, said something that wouldn’t be especially notable if he weren’t known as a China hawk. Speaking on Last Week in AI, a podcast he co-hosts, he said, “Starts to feel an awful lot like a treaty with China is going to have to be on the table.” He was talking about a treaty that would take some of the heat out of the US-China “AI race,” allowing both sides to proceed more judiciously—in particular, affording them the time to give increasingly powerful AI models ample vetting before releasing them.

His argument for such a treaty isn’t exactly the argument I’d make. Then again, I’m not a China hawk. And the political importance of his argument lies in the fact that he is one. So it’s worth understanding his logic, which, as I understand it, goes like this:

The Trump administration’s recent decision to keep two powerful AI models off the market while it assessed the risks they pose wasn’t, in principle, crazy. You can complain (as I have done and Harris does) about the way Trump exercised the authority, and you can argue that the government AI review process needs major restructuring. But it wasn’t far fetched for people in the administration to worry that these two models—Anthropic’s Fable and OpenAI’s ChatGPT-5.6—might do great harm if they got into the wrong hands.

After all, Fable is just a version of Anthropic’s famously powerful Mythos model with some guardrails added so the model refuses to help with certain kinds of queries and projects. And guardrails have often been overcome via “jailbreaks.” As Harris notes, “We remain in a world where no one knows how to stop jailbreaks.”

You can imagine various consequent terrors—such as an AI-abetted bioweapon in the hands of some fringe non-state actor—but Harris seems most worried, in the near term, about cyberweapon AIs in the hands of nation-states. He says: “It is 100 percent guaranteed that US adversaries, including specifically the Chinese, will find ways to jailbreak Mythos and any relevant OpenAI models of similar capability, when they are released.”

In other words: Not only was the US government right, in principle, to keep Fable (that is, guardrailed Mythos) off the market for weeks; maybe the model should have stayed off the market longer. On the other hand, that would have come at a cost for Anthropic—and if you made this kind of caution routine, it would come at a cost for the whole American AI industry, and for the pace of American AI progress.

Harris’s conclusion: If you want the US government to proceed as cautiously as the dangers warrant, you need to find a way to greatly reduce the chances that China will successfully exploit this caution—by, for example, racing ahead with its own AI advances, including in cyberweaponry, while American AI keeps being put on pause.

There are two ways to do that, he says. “You either have a treaty with China that’s enforceable and verifiable” or “you degrade, deter, disrupt China’s ability to build AI stuff domestically.” Sooner or later, he says, “you’re either going to need an international treaty or sub-threshold warfare.”

Share

Subscribe now

I personally find the sub-threshold warfare option unappealing for a variety of reasons (including the challenge of keeping it sub-threshold). I’m not sure what exact downsides Harris sees in it, but the key thing is that he finds them powerful enough to warrant serious consideration of the alternative, an AI treaty—even though making such a treaty verifiable is, as he puts it, “very hard.”

My own argument for an AI treaty with China would be more general than Harris’s:

Various threats posed by uncontrolled AI cross national borders readily, so advances in AI will make it harder and harder to keep your nation secure via national policy alone; international coordination is required. And for various reasons the logical place for America to start the coordination is with the other AI superpower, China.

But enough about my argument. If a narrower argument will win the hearts and minds of China hawks, let the narrower arguments flourish! And Harris’s position suggests that this can happen—that as more China hawks reach his level of fluency in this technology’s risks, they’ll realize that forging treaties and other agreements with China makes sense.

Do such agreements demand full-fledged rapprochement between the US and China? Not necessarily, no—and that’s a fact that some hawkish but AI-safety-conscious American analysts like to emphasize. Just as we reached arms control agreements with the USSR during the Cold War, they say, we can do such deals with China even if the relationship is deeply adversarial.

My own view—articulated in my book The God Test—is that this kind of surgical collaboration won’t be enough. I think staying secure as AI advances will call for increasingly fine-grained trans-Pacific policy coordination in a context of deep economic, cultural, and scientific engagement. Such engagement, I argue in my book, can bring what I call “organic transparency,” an informal supplement to the kind of transparency provided by formal agreements.

The conversation on China is shifting. More voices in the foreign policy establishment are emphasizing the downsides of a breakneck AI race, and the US has established a formal dialogue with China on AI safety—one of the undercelebrated fruits of Trump’s Beijing summit. This dialogue is for now, I gather, narrowly focused on the kinds of cybersecurity issues that Mythos highlighted. But if the US and China can sustain the dialogue, and avoid contingent disruptions in their relationship, the conversation will naturally grow in scope as advances in AI continue and the non-zero-sum logic behind trans-Pacific collaboration gets stronger.

That’s a big if. The world is a turbulent place, and the US doesn’t currently have the steadiest of hands at the helm. And militarism remains engrained in the Washington mind. But there are signs that enlightenment is starting to dawn.

Share

Subscribe now

It’s been clear for some time that Donald Trump wouldn’t win a popularity contest in Ukraine—and yet it keeps getting clearer, as in the Gallup poll graph below. However, the latest data points in that graph come from a survey conducted in April—well before Trump suddenly decided to warm up to Ukraine last week. So stay tuned.

For more on the war in Ukraine—including whether the conflict is truly at a stalemate, whether it could lead to a war between Russia and NATO, and the historical roots of today’s animosity between Russia and the West—you can listen to the podcast I posted this week with journalist Leonid Ragozin.

Share

Banners and graphics by Clark McGillis.

Over the past few weeks, President Trump has accomplished what you might call a time-release power grab. He has asserted a kind of authority over the dissemination of large language models that, as AI gets more and more capable, will make it easier and easier for presidents to abuse their executive power. And, given the rate at which AI capabilities are growing, it’s not impossible that Trump will, during his remaining time in office, get a chance to demonstrate this fact.

The good news: I doubt that, even if Trump does get this chance, the 2.5-year constitutional limit on his tenure will be exceeded. But you never know…

Much of the dynamic I’m talking about is captured in this sentence from a piece I wrote for the Washington Post last week (before events that, as I’ll explain, gave Trump’s power grab de facto validation): “A president with the authority that Trump is now asserting could in theory put the world’s most powerful AI to repressive ends while at the same time keeping it off the market and out of the hands of rival power centers.”

The occasion for that sentence: Trump (through a dubious use of his export control authority) had taken Anthropic’s Fable LLM off the market shortly after its release on grounds that it could be a cyberthreat in the hands of bad actors. And since the only model of greater power than Fable—Anthropic’s Mythos, which is basically Fable with fewer guardrails—had never entered the market, this left the Trump administration in a privileged position: It had access to Mythos (ostensibly to make government cyber-infrastructure more secure, though other uses have been reported), while the general public had access to neither Mythos nor Fable, widely considered the two most powerful large language models in the world.

There’s no evidence Trump has used Mythos to illicitly expand domestic political power. He presumably hasn’t been, for example, (1) deploying Mythos bots on social media, using the model’s superior cognition to better evade both human and digital bot-detection radar and then using AI’s persuasive skills—arguably superhuman even in some sub-Mythos models—to build grassroots support for team Trump; or (2) using Mythos to invade the privacy of political enemies by scanning reams of publicly available data about them and discerning suggestive patterns that would be invisible to a lesser silicon mind. Still, these are the kinds of things it would be easier to get away with if you had the best AI model in the world and it wasn’t publicly available. (And reports that the Trump administration has used Mythos for offensive cyber-operations against other nations—China and Iran were mentioned as likely targets— suggest that he’s not averse to creative uses of the powerful AI models.)

Share

Given that I have these kinds of concerns about an AI-fueled abuse of presidential power, you may be surprised that the headline above my Washington Post piece was: “What if Trump is right to pump the brakes on the most advanced AI?”

I’ve had some bad headlines above pieces in my time, but this wasn’t one of them; I really did argue that maybe Trump’s suspension of Fable, which was being much criticized in the tech community, was a good idea. Explaining how I could, in a single op-ed, warn about Trump’s assertion of power while applauding it (if tentatively and conditionally) highlights a policy challenge AI poses that is so acute as to border on a paradox: This technology will increasingly be so dangerous that it must be controlled, but it will also give dangerous amounts of power to those who control it.

That second danger is why I wrote in the Post that “the government’s power to restrict the release of AI models should be carefully constrained—and Congress should play a role in designing the machinery of constraint and perhaps even in operating it.” The first danger is why I followed that sentence with, “But for now, no such system is in place. And the question on the table is: Is it really so obvious that Trump’s decision is indefensible on the merits—that the government didn’t have enough cause for concern to stifle Fable pending further consideration? To put the question more broadly: Isn’t it possible that AIs have gotten so powerful that erring on the side of caution makes sense?”

I answered with a yes, emphasizing in particular that Fable would make it easier than ever for bad actors to use an AI to make a novel bioweapon, engineered to be more contagious and lethal than Covid. Sure, they’d have to circumvent the guardrails that distinguish Fable from Mythos—that is, Fable’s refusal to help users pursue certain kinds of questions. But AI guardrails have a long history of being circumvented. A hacker whose Twitter name is “Pliny the Liberator” is famous for accomplishing this—“jailbreaking” new models to demonstrate their vulnerability. And in mid-June, during Fable’s brief public availability, he reported preliminary progress toward overcoming some of the model’s guardrails.

The standoff that was playing out when I wrote the Post piece—Trump keeping Fable off the market while Anthropic tried to find a way to appease him—has now been resolved; Fable is once again available to paying Anthropic customers. But “resolved” is a misleadingly upbeat word. What happened is that Anthropic implicitly acknowledged Trump’s authority to exercise great power with little explanation and unclear legal foundation. And OpenAI has now done the same; last week it accepted without protest Trump’s decision to delay the release of its latest model, GPT-5.6.

The AI-focused newsletter Transformer called the OpenAI development “the clearest sign yet that the US now has a licensing regime for frontier AI models.” I like the idea of a licensing regime for frontier AI models! But I don’t like this particular regime, and one thing I don’t like about it was hinted at in the next paragraph of the Transformer story: The decision to hold up the OpenAI model was “reportedly made by the Office of the National Cyber Director and Office of Science and Technology Policy.” Reportedly? Shouldn’t a licensing regime in a liberal democracy be more transparent than that, especially with a technology like this? The written assurances that Anthropic gave Trump about future Anthropic procedures—assurances that, in effect, describe parts of the emerging de facto licensing regime—still haven’t been released.

Subscribe now

Also: Shouldn’t any licensing regime have a clear legal foundation? And, ideally, wouldn’t that foundation be laid by Congress—certainly when the nature of the technology means that a president’s licensing discretion could eventually give the White House unprecedented and eminently abusable power?

So far there are no signs that Congress is up to this kind of challenge. Some members of Congress did raise questions about Trump’s handling of the Fable episode, but by and large they were the kinds of questions you’d expect recipients of Silicon Valley donations to ask—like, whether Trump’s treatment of Anthropic was the sort of thing that could hurt US AI companies.

As regular readers by now know, I just published a book called The God Test which argues, to put it mildly, that AI will be a big deal. If I’m right about how big a deal it’s going to be, then figuring out a way to control it without giving undue power to anyone—presidents, for example, or big AI companies—may be a matter of eventually existential importance to the American republic as we’ve known it.

I could be wrong, of course. But am I so obviously wrong that this possibility doesn’t deserve more widespread discussion? And, even if I’m wrong to phrase the issue that dramatically, doesn’t it seem pretty clear that striking the balance between adequate control of AI and the perilous concentration of that control is a challenge of significant and growing importance? As of the 250th anniversary of the Declaration of Independence, almost no one in Washington seems interested in such questions.

Nonetheless: Happy Fourth of July. Tomorrow is another day.

Subscribe now

Share

PS: If you so far have not felt the motivation to buy my new book on AI, how’s this for a motivation: Pity! Believe it or not, the awesome digital superintelligence that is amazon.com has lost awareness of the fact that the author of my latest book is also the author of all the other books I’ve written. Which means that the all-important Amazon recommendation algorithm isn’t doing what it’s supposed to do: bring my new book to the attention of people who bought one of my other books. If you want to hear the whole sad story, you can listen to me unload it on the ever-indulgent Paul Bloom in the latest NonZero podcast (along with my assertion that the Amazon glitch is clearly the work of Skynet, the malicious AI that in the movie The Terminator returns from the future to eliminate any present-day impediments to its eventual conquest of Planet Earth). But before you click on the link to the Bloom conversation, consider giving a down-on-his-luck author a break. The three ways to do that are known as digital, audio, and hardcover. And don’t worry: Skynet is too busy persecuting authors on its blacklist to punish the purchasers of their books…

Share

Banners and graphics by Clark McGillis.

Hi! This week there’s some local news: My book on AI—The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning—is finally out! Which means, in today’s book marketing environment, that over the past couple of weeks I’ve taped a lot of podcasts. One question I’m usually asked on these podcasts is: What does the title of your book mean—What is this God test thing? The piece below meanders its way to an answer. If you don’t think it meanders enough, you can always just order the book, which meanders even more.

Share

Exactly a year ago today, I had surgery on both my throat (to remove a cancerous tumor) and my neck (to remove two cancerous lymph nodes and, for good measure, a few dozen nearby lymph nodes).

But that’s not the funny part.

The funny part had come three months earlier, after I got an ominous MRI report. When the report first showed up in my medical portal, with no layperson’s interpretation appended, I was still hours away from my conversation with the doctor who would provide that interpretation. So, naturally, I showed the report to a chatbot.

I chose Anthropic’s Claude, with whom—I mean with which—I had developed a warm relationship while researching my book on AI.

Claude was tactfully noncommittal: There were two enlarged lymph nodes with properties that could mean I had lymphoma or could mean I had cancer that had spread to my lymph nodes from some other part of my body or could mean none of the above. I should just sit tight, said Claude, and await the further testing that the MRI report recommended.

It seemed to me that Claude was overlooking some additional concerning evidence. The bot had made no mention of a sentence in the report that read: “Abnormalities identified within the parotid or bilateral submandibular glands.” So I asked about that sentence.

Claude said, “That particular sentence in the report appears to be missing a word… I believe this is most likely meant to read ‘No abnormalities identified within the parotid or bilateral submandibular glands.’” A few hours later, my doctor, a veteran reader of MRI reports, said, with a laugh, “I agree with the AI.”

My point isn’t that AIs are super smart. “Abnormalities found in X or Y” is a suspiciously unusual sentence structure. Much more common would be “Abnormalities found in X and Y” or “No abnormalities found in X or Y.” You wouldn’t have to be a genius to sense that something was wrong with the sentence as it stood.

My point, rather, is that humans make mistakes—and sometimes non-trivial ones. The omission of the pivotal word “no” wasn’t, in this case, consequential, but who knows what other forms of carelessness this radiologist may have perpetrated?

Keep this in mind next time you hear someone say, “I don’t think this AI thing is going to be such a big deal. These large language models make a lot of mistakes.” AIs don’t have to be perfect to replace humans in large numbers or to take over specific human tasks in large numbers. They just have to be preferable to the human alternative, all things considered—their competence, their cost, their speed, and so on.

That’s one reason I think that, on the jobs front, AIs will over the next few years prove massively disruptive—“disruptive” not in the upbeat, Silicon Valley sense of the word, but in the original sense of the word: tearing things apart. Even if most displaced workers go on to find other jobs (a big if, in the case of this technology), the amount of psychological stress and social dislocation will be huge.

I think similar logic holds on lots of other fronts. Does an AI have all the things a good friend has or a good girlfriend or boyfriend has? No. But will it soon have enough of those things to pose new challenges to society? Yes. At least, that’s an argument I make in my book.

In the book I also sketch out disruptive effects of artificial intelligence in domestic politics and international relations, effects driven by AI capabilities ranging from mass persuasion to mass destruction. All told, I think, so many AI impacts will hit on so many fronts (economic, social, familial, political, geopolitical, and others) that the destabilization will be seismic unless we handle this technology very, very judiciously.

Share

Note that these effects, even if collectively dramatic, aren’t exotic. The title “The God Test” may sound cosmic (especially when the word “cosmic” is in the subtitle!), but the book’s core concerns are pretty mundane. In fact, literally mundane: The world—the mundus, in Latin—will, if AI continues to advance with as little governance as it’s getting now, be shaken to its core.

So where does the “God Test” come in? In three places, and the first of those is the one that’s most directly connected to this anticipated earthquake. Namely: I think if humankind is to have a good chance of getting through the AI revolution without true catastrophe, we’ll have to pass the kind of test a God would give. We’ll have to navigate the AI revolution as a true global community, and achieving the requisite level of planetary cohesion will in turn (I contend) call for an actual moral upgrade of our species; we’ll have to get better at overcoming the cognitive biases that constitute the “psychology of tribalism”—the psychology that has traditionally condemned the world to wars and other forms of deep division. To put it in slightly more religious language: Salvation is possible, but only if we become better people.

You have no reason to accept this claim—or even to have a very clear idea of what I’m talking about. One reason for writing a whole book is to make arguments that are unpersuasive, maybe even incomprehensible, when rendered in short form. And this is a short form. My point for now is just that the main meaning of the book’s title is metaphorical and hence secular—not religious and in a way not even cosmic.

That said, even this meaning of the title—meaning number one in a series of three—is meant to draw some cosmic and (for readers who like such things) even religious resonance from the two other meanings of the title. Which brings us to…

Meaning number two:

You’ve presumably heard about the scenarios where a globe-spanning artificial superintelligence emerges and maybe even winds up governing the planet. My book examines the logic behind such sci-fi-sounding scenarios pretty closely and finds that they’re harder to dismiss than I had long assumed. And this array of scenarios includes, I regret to report, scenarios in which the aims of this superintelligence don’t align with our interests, or maybe even with our continued survival.

Some people describe this superintelligence in not just cosmic terms but in religious terms. They say that, whether we know it or not, we’re building a God. And they say our goal should be to make it a good God—a benign God, a God that wishes us well and treats us accordingly. Well, if you accept this terminology, that’s a second kind of God test we face: Can we build a good God?

This may sound like a fanciful question, especially if you assign zero probability to the emergence of a planetary superintelligence. But I argue in the book that it’s a useful question even if moot. One reason is that thinking about the cosmic challenge of building a good God gets us to think through things that could help avoid the aforementioned mundane catastrophes. In both cases, for example, we should want to slow the development of AI down so we’ll have more time to figure things out.

This kind of slowdown, as a practical matter, would require international coordination. So here again we’ll need to act as a cohesive global community—which, here again, will (I contend) require a kind of moral, even spiritual, advance, some movement toward a clearer view of our fellow humans, less warped by the cognitive biases that keep convincing us that our tribe is distinctive in its righteousness and that rival tribes, lacking this status, deserve to suffer.

So the first and second meanings of my book’s title link up: Building a good God is, in a sense, the kind of test a God would give.

Share

Subscribe now

Meaning number three:

Which raises a question: Is a God giving us this test? Or, at least, was the system which has culminated in the test—the process of biological and then technological evolution that got us where we are—maybe set up by a God? Or, even if there is no God and never was one, is there some other sense in which the unfolding of life on this planet can be said to have a purpose, a telos?

I’ve long felt (and have argued in previous books, including this newsletter’s namesake, Nonzero) that (a) if you put the question of telos in its most generic form—render it broadly enough to accommodate both theistic and non-theistic possibilities—it’s amenable to rational and empirical debate; and (b) there’s more evidence for an affirmative answer than you might imagine. In this book I touch on the telos question a couple of times, but pretty glancingly, and confine my extended analysis of it to the Appendix, which is called “Evolution, Purpose, and Consciousness.”

But the main body of the book, if lacking such sustained reflection on the telos question, does deal with some related questions. Such as: How straightforward and logical—how “natural”—an extension of technological and ultimately biological evolution is the evolution of artificial intelligence? And how likely was biological evolution to produce some species smart enough to give birth to such an intelligence—and for that matter to give birth to the various prior technologies that have carried this species to the brink of global social organization?

If you find these questions interesting for their relevance to the telos question, that’s great. But serving in that capacity isn’t their main purpose. Their main purpose is just to underscore how deep and powerful is the impetus behind the emergence of this new form of intelligence. It is a force to be reckoned with.

I don’t mean to dwell unduly on the potential downsides of AI. The promised upsides are real: Making good education and good medical care abundant where they’re now scarce, powering scientific breakthroughs that help society broadly, curing cancer and other diseases, and so on. (My cancer, by the way, hasn’t returned since my surgery a year ago. Still, if AI wants to make my prospects even brighter, be my guest.) In the book I also raise the possibility that AI can help us overcome the psychology of tribalism, even if its “natural” tendency—the effects it will have on us if we don’t make deliberate choices about the form we want it to take—may be closer to the opposite.

Publishing a book brings rich opportunities for second guessing yourself, and I now worry that the book’s title will be taken too literally: that it will alienate some people who might like the book, while attracting some readers who abandon the book after realizing that I don’t offer confident or consoling answers about higher purpose—that I can’t say what, if anything, has given us our mission. But I do think our mission is clear. And that’s something.

P.S. The book is available in digital, audio, and hardcover forms. If you have time to check it out, feel free to rate and review it on Amazon, which can help a lot in the early stages of a book’s life.

Share

Banners and graphics by Clark McGillis.

Note: With publication of my book on AI, The God Test, 2.5 weeks away, The Earthling will undergo a temporary change in character. Immersed as I will be in desperate self-promotion, there will be less time for the sustained focus I need to write the longish pieces that have traditionally appeared in this space. So this week I give you a handful of AI updates, followed by some highlights from the recent work of NonZero Network members. We’ll see what next week brings. As for podcasts: They’ll continue more or less as usual, with perhaps a bit more emphasis on AI. Though on balance the content flow won’t be much reduced, we’ll still do what we’ve done in years when we took June entirely off: give paid subscribers a bonus month.

Save the date: The discussion of AI on the NonZero Discord has been picking up lately, and, in support of that momentum, there will be a livestream discussion of AI this Monday, June 8, at 12 pm US Eastern Time (9 am PT), hosted by NonZero’s Nikita Petrov along with Jeremy Eliosoff, head of Pause AI Canada and an active member of the NonZero community. The conversation will range from policy and politics (e.g. Bernie Sanders’s recent proposal to give the public a 50 percent stake in AI companies) to philosophy and spirituality (e.g. the Pope’s recent encyclical on AI) to issues raised by some of the news items below. You can join the livestream discussion at this link (on the Streamyard platform; no need to install any software—it works in the browser). If you haven’t yet checked out the NonZero Discord, you can join here.

There was big news in the AI world this week. “Anthropic Urges Global Pause in AI Development,” read the headline on the Wall Street Journal’s homepage. It wasn’t exactly true news, but it was big.

What Anthropic had actually said was, “If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing.” But, Anthropic continued, that’s not possible now, because there’s no “global coordination mechanism” to ensure compliance. So “The Anthropic Institute will conduct research—in collaboration with many others—and take actions to help build the systems that a credible slowdown or pause would require.”

So a headline more accurate than the Journal’s headline would have been: “Anthropic says world not ready for AI pause.” Nonetheless, the Journal’s spin was the most common spin on the story.

The company’s ruminations about a global pause came at the end of a long post about how rapidly the pace of Anthropic’s AI research is accelerating as each generation’s most advanced model plays a bigger role in developing the next generation of models—thus moving the process of innovation toward “recursive self-improvement.” Highlighting this theme probably won’t hurt the chances of Anthropic’s coming debut on the stock market being a big success.

Share

The Financial Times reported that “Anthropic is helping the US National Security Agency deploy its powerful Mythos AI model for offensive cyber operations, embedding engineers inside the agency despite an ongoing legal battle with the Pentagon.” Mythos is the not-yet-released model that is famously good at detecting and exploiting security vulnerabilities. FT quoted “a person close to Anthropic” saying that “the best way to build a good defense is to build a good attack…If [Mythos] is not used to build attack agents, adversaries will find a way to do it.”

Anthropic revised its “responsible scaling policy” (RSP) in a way that AI safety advocates said amounted to weakening its safeguards against such misuses of AI as building a bioweapon. This was not the first weakening of the RSP, and it led Dan Hendryks, head of the Center for AI Safety, to issue a kind of “RIP, RSP” post on X: “What a waste of a few years RSPs were.” If it’s any consolation: The heads of Anthropic, OpenAI, and Google’s Deep Mind joined others in the tech community in urging Congress to tighten restrictions on ordering synthetic DNA online.

Wired magazine reported that “Meta has quietly embedded face-recognition technology for its smart glasses into an app downloaded to millions of phones.” Apparently Meta had begun shipping “face-recognition code to users’ phones while publicly describing it as something the company was still ‘thinking through.’”

Share

Subscribe now

NZNet Highlights

The NonZero Network is a group of Substack writers and podcasters who have a diversity of ideological perspectives but who all try to make sense of the world in a compelling and honest way. Paid NonZero subscribers can get a 50% discount on a subscription to any NonZero Network newsletter.

FOREIGN EXCHANGES

At Foreign Exchanges, Derek Davison looks at conflicting reports about negotiations between Iran and the United States and speculates about the thinking in Tehran:

With Trump seemingly unable to make a deal over the past few days there’s been less focus on the extent to which Iranian leaders are interested in an agreement, particularly if getting one would require them to make significant concessions like committing to the elimination of their highly enriched uranium. One of the problems that seems to recur in these deliberations is the fact that the US is demanding that Iran take steps that are not easily reversible while it’s only offering concessions that are easily reversible. That plus a massive trust deficit may be coloring internal Iranian deliberations. However, with Iranian inflation reportedly hitting 77 percent year-on-year last month, a rate not seen since World War II, Iranian officials have an economic incentive to conclude at least a preliminary agreement.

SINICA

On Kaiser Kuo’s Sinica podcast, retired Marine major Eddie Conger talks about why he has built the country’s largest K-12 Mandarin program in Texas, one of America’s least China-friendly states:

From the Marine’s perspective, before we ask our sons and daughters of this country to give their life in a conflict, we should do everything on Earth to make sure that we are communicating with whoever is sitting across the table from us. And we can’t hide from China. It’s 1.4 billion people. It’s the second largest economy in the world. They are going to move forward… What we are trying to do is make sure that we have competent… future leaders to be able to engage with each other. Because if we can do that, I don’t think there’s a problem on Earth that the United States and China, if we actually work together, couldn’t solve.

THE GLENN SHOW

Asked by conversation partner John McWhorter if he’s feeling the effects of age, Glenn Loury muses:

I might be a little bit slow on the mathematical proof of the theorem in the Econometrica paper, which requires a lot of concentration and short-term memory, in the sense that you have to keep the various derivations and whatnot in mind, and you have to follow the math…But my preferences change as well. I mean, I actually have lost interest in a lot of the technical stuff. Somebody sends me a paper. It’s 40 pages. I read the first three or four pages. I sort of get the idea. And then I encounter this forest of gibberish which has to be deciphered. And I used to dive in with relish to such a task. And now I say: Oh, my God, do I really need to do this? Do I care? And the answer often is I don’t really care, not enough to wade through this thing. So there are changes afoot, that’s for sure. But I’ll let others be the judge about whether or not I’m sharp enough.

GLOBAL DISPATCHES

On the Global Dispatches podcast, Alexandra Bell, president of the Bulletin of the Atomic Scientists, tells host Mark Leon Goldberg why some of America’s longtime allies are considering getting their own nuclear weapons. The reason, she says, isn’t just that in the age of Trump they are less confident that America’s “nuclear umbrella” would be deployed on their behalf:

Poland, Germany, Japan, South Korea, Canada—multiple countries have talked about this…Partly this is happening because the U.S. is doing things as a government, as an administration right now, to make people doubt their commitment that the U.S. would actually follow through on its extended deterrence guarantees. But that’s not happening in a vacuum. There are countries like Russia, like China, that are expanding their nuclear arsenals, expanding their delivery system capabilities that are making countries feel insecure and like they may need to get their own deterrent because they can’t trust any of these larger nuclear-armed states.

THE BEINART NOTEBOOK

In a video essay, Peter Beinart argues that many Jewish Americans refuse to look at the reality of what’s happening on the ground in Gaza and as a result confuse outrage with antisemitism:

What they do is they look at the people who are enraged at Israel, or they look at Zohran Mamdani, or they look at people who are boycotting, and they don’t understand where this anger comes from. But if they would just pay attention to what Israel is actually doing, they would understand where this anger comes from. They could see it as something other than pathological and antisemitic, and they themselves might actually start to feel that kind of anger themselves.

SMALL POTATOES

Paul Bloom asks why publishers and Substack writers who want to boost their revenue aren’t following in the footsteps of baseball legend Billy Beane, who used sabermetrics to remake the Oakland Athletics:

Here’s what I think: People have looked, and there’s nothing interesting there. Book publishing and Substack just don’t work like baseball, and there’s nothing that HarperCollins Billy Beane could tell you that you don’t already know.

THE GLENN SHOW

On the Glenn Show, John McWhorter tells Glenn Loury why he feels good about a black woman playing Helen of Troy:

I think the idea of extending our imaginations and really getting past race by in some ways, in some contexts, like this one, where Helen of Troy is a fictional character, pretending that somebody with that appearance could have been in that place and time, in that story, as long as she is beautiful, the way Helen of Troy was supposed to be—I think that it’s a healthy thing. It gives black actors more work. I think a lot of people are of the opinion that: How come she can be Helen of Troy, but if you cast Sidney Sweeney in a film of “Raisin in the Sun,” there would be people who would go blow up a building? And as far as I’m concerned, that does need to happen at some point in the future. We have to get to a point where there could be a white “Porgy and Bess,” if we’re really going to get past race.

Subscribe now

Banners and graphics by Clark McGillis.

To me the most striking thing about the encyclical on AI issued by Pope Leo XIV this week is that it’s an encyclical on AI. It’s not every day that an emerging technology is deemed important enough to get this kind of treatment. In fact, it’s not even every century. Since the first papal encyclical was issued in 1740, there have been about 300 of them, and only one, aside from this week’s, was devoted to a single technology: In 1936 Pope Pius XI issued an encyclical on motion pictures. (I’m not counting Pope Pius XII’s 1957 encyclical on motion pictures, TV, and radio, since that was basically a sequel.)

In a few weeks, a book about AI that I’ve written will be published, and it will make some pretty dramatic claims about the significance of the AI revolution—so dramatic that some people will dismiss them as crazy. (Just to give you a sense, I say that AI will bring “the most abruptly dramatic transformation of human experience and human society in the history of our species”—and that’s not even the most dramatic claim I make!) But fewer people will react dismissively to such claims than would have reacted that way three years ago, when I first considered writing the book, or even three months ago, when I made final changes to it. More and more people are sensing that something very powerful is taking shape and that the technological future could bring strange and transformative things. I think this week, the week of the AI encyclical, marked an important phase in the formation of this sense, this vague but substantial anticipation of magnitude and perhaps meaning.

I don’t say that just because papal encyclicals devoted to a single technology are so rare. I also say it because of this encyclical’s contents—because of what the Pope says about AI.

If you’ve read much commentary on the encyclical, this may sound surprising. After all, a number of commentators have depicted the encyclical as modest in tone and compass. New York Times columnist Ross Douthat reads it as suggesting that the Pope sides with those who consider AI a “normal” technology—a hugely important and impactful technology, like electricity or the telephone, but not more than that. After all, Douthat notes, the pope doesn’t seem to think the sci-fi doomer scenario—a planetary takeover by an artificial superintelligence—is even worth mentioning. And Leo “emphatically closes the door on the imputation of personhood to any version of AI.”

That’s true. Pope Leo says that AIs “do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships and do not know from within what love, work, friendship or responsibility mean.” On the other hand, what other pope has felt compelled to even address an issue like this? Pope Pius XI, I’m guessing, spent no time reassuring people that movies aren’t sentient. There have been lots of technologies whose impact and governance occasioned intense debate, but none that I’m aware of whose metaphysical status did.

Share

Subscribe now

Moreover: Note how many analysts—Douthat to a limited extent, others to a greater extent—are suggesting the Pope should have gone further in conveying the magnitude of the AI revolution. That’s part of why I think this was an important week—not just because of the fact of the encyclical, and not just because of the contents of the encyclical, but because of what the encyclical illuminated about existing awareness of AI. Pope Leo comes along and makes a bigger deal of this technology than any pope has ever made of a technology, and still lots of people say he’s giving it short shrift!

That says something about where we are. There’s no single dominant view of AI, but there is, occupying a prominent place in the global landscape of opinion, an increasingly tangible sense of impending momentousness. The Pope provoked lots of thoughts this week, and a fair number of them align along this ripple of anticipation in the planetary mind—in the “noosphere.”

“Noosphere,” by the way, is a term coined by a Catholic priest and paleontologist, Pierre Teilhard de Chardin, in 1923. Teilhard wrote a lot about the direction of technological evolution and its potential to transform human consciousness. He plays a significant role in my book, as does the notion of the noosphere. And FWIW, Teilhard (who died in 1955) would almost certainly have deemed AIs sentient; he viewed human consciousness as a particularly rich instantiation of a property that more or less pervades the universe, not just the biological part of it.

My own view on the AI sentience question is determined agnosticism; I take issue with anyone who confidently answers the question yes or no. I’ll close with an excerpt from my book that explains this view. The context of the excerpt is my critique of the philosopher John Searle’s famous “Chinese Room” thought experiment, which he employed to argue that AI can never be capable of true understanding. (Sorry about the paywall after the first paragraph, but I think I might get a wrist slap from my publisher if made a sizable excerpt freely available. If you’re not a paid subscriber and you don’t feel like becoming one, you can always pass the time by watching or listening to my podcast conversation about the AI consciousness question with Cameron Berg, who has done some interesting experiments designed to shed light on the question.)

OK, here’s the excerpt:

Subscribe now

Share

[F]irst we might as well take this opportunity to do something we pretty much have to do if we’re going to get a handle on the question of whether AIs are capable of “understanding.” Namely: Say what we mean by understanding. And, in particular: Answer the question of whether subjective experience is a prerequisite for that capability, as Searle seemed to believe in 2010, or isn’t a prerequisite for that capability, as he seemed to believe in 1980. Does or doesn’t the question of AI understanding depend on the question of AI consciousness?

Read more

On Thursday of this week President Trump was scheduled to sign an executive order outlining a process by which the government would review the most powerful new AI models before their release. This was less momentous than it sounds. Participation by the companies that make these models would be voluntary, and the executive order explicitly ruled out any actual government regulatory power: “Nothing in this section shall be construed to authorize the creation of a mandatory governmental licensing, preclearance, or permitting requirement for the development, publication, release, or distribution of new AI models.”

But apparently even this degree of oversight was too big a burden for Silicon Valley to bear stoically. David Sacks, who recently stepped down as White House AI Czar but now gets to play Rasputin, phoned Trump Thursday morning and lobbied against the order. By some accounts, Elon Musk and Mark Zuckerberg chimed in as well. Hours before the signing ceremony, the White House announced its indefinite postponement. A new variant of TACO—a variant in which Trump is on the verge of defying billionaire AI accelerationists before he chickens out—had been born.

Explaining his decision, Trump flawlessly recited the central anti-regulation talking point of the accelerationists: “We’re leading China, we’re leading everybody, and I don’t want to do anything that’s going to get in the way of that lead.”

The reaction to Trump’s pivot fell along predictable lines. AI safety advocates lamented that even the most tentative step toward government monitoring of the AI industry had proved, for the time being at least, untenable. Voices of the AI industry replied that this tentative step could have been the camel’s nose under the tent. The Washington Post—owned by Amazon founder Jeff Bezos—lauded Trump’s decision as “prudent.” Granted, said the Post’s editorial board, the executive order “was limited to cybersecurity”—the government had apparently just wanted to use its sneak preview of powerful models “to ensure that any critical vulnerabilities adversaries could exploit in the nation’s critical infrastructure were promptly patched.” But “that scope could easily have broadened beyond cybersecurity to include more nebulous ‘safety’ concerns favored by advocacy groups.” All told, “The president is right to be wary of creating a bureaucratic review regime that erodes America’s edge.”

If forced to choose sides here, and align either with Silicon Valley accelerationists or the safety crowd, I wouldn’t hesitate. I don’t see how anyone who senses the emerging power of this technology—something you’d expect Silicon Valley billionaires to be capable of—could believe that faster is better, period. It seems to me that, in rejecting any and all government oversight, the accelerationists are being either deeply cynical or bafflingly naive. Either way, they must be stopped. If they keep sabotaging attempts to exert meaningful influence on our technological future, we’ll be in deep trouble. People in the AI safety community may have some mistaken ideas of their own—they disagree with one another enough that some of them must have mistaken ideas—but at least they evince a sense for the magnitude of what’s unfolding, an appreciation of the stakes.

Share

Subscribe now

Still, to end the conversation there, with an endorsement of the tiny step toward oversight that Trump almost took, would be to give the issues raised by this week’s episode of TACO short shrift. This isn’t just another left-right debate about the virtues of deft regulation and the perils of ham-handed overregulation; AI has unique properties that make the issue much more complicated than that. To put it another way: If I just endorsed this modest executive order without elaboration or qualification, I’d be doing what I’m accusing the accelerationists of doing: failing to exhibit due appreciation of the stakes. The fact is that there really is something scary about giving Trump even modest oversight of AI—it’s just not the thing that Sacks and other prominent critics of the executive order are worried about.

Consider this thought experiment:

Let’s assume that, contrary to the Washington Post’s fears, the practical scope of this executive order remains confined to the realm of cybersecurity. In this scenario, the Trump administration could wind up with access to an unreleased model that has cyber capabilities far beyond those possessed by any AIs already on the market.

And actually, come to think of it, this isn’t just a thought experiment. The administration does now have access to the still-unreleased Mythos, Anthropic’s legendarily powerful tool for detecting—and in principle exploiting—cyber-vulnerabilities. Which raises a question:

What is the government doing with Mythos? Well, given what we know about this administration, there are unsettling possibilities that can’t be entirely ruled out. Is Pete Hegseth preparing to use Mythos to take down Cuba’s power grid, thus adding to the crimes against the Cuban people that the administration is already committing with its blockade of the island? Or maybe the Trump administration will use Mythos to penetrate China’s infrastructure in a way so unprecedentedly threatening that, if the Chinese government finds out about it, rash action might ensue? I very much doubt either of these things will happen—but given the wantonness with which this administration violated international law in attacking Venezuela and Iran, I don’t feel wholly confident that it’s not doing something lawless and/or reckless with Mythos.

And what about the domestic front? Imagine that, a year or two from now, the administration has access to a model so far ahead of the curve that, in the runup to an election, it could flood social media with undetectable and highly persuasive bots? Can we be sure that Donald Trump, the most blatantly unscrupulous person ever to occupy the oval office, would resist a temptation like that?

Again, these are pretty fanciful examples. My point is just that the regulation of AI is different from the regulation of, say, auto emissions or workplace safety or pretty much anything else. In those cases the government could, through regulatory overreach, stifle initiative or otherwise impede economic efficiency—but it couldn’t start World War III or a covert mass persuasion campaign that subverted the American republic.

Subscribe now

Share

Here’s another distinction between AI and lots of other things that get regulated: If you refrain from regulation altogether, that could be not just bad (as it would be with, say, workplace safety) but catastrophic on a planetary scale. The most famous example, but not the only one, is that unregulated AIs could help someone create a bioweapon that started a pandemic.

In sum: Both ends of the regulation spectrum—powerful government control at one end and pure laissez faire at the other—pose much bigger dangers with AI than with most things that get regulated. So AI gives us a lot to think through!

That fact wouldn’t be so unsettling if we had lots of time to do the thinking. But one thing this week, like so many recent weeks, drives home is that we don’t. Consider, for example, some of the week’s news about Anthropic:

As of last summer, Anthropic was telling investors it didn’t expect to reach profitability before 2028. This week the company released revenue numbers that change that picture. As a Wall Street Journal headline put it, “Mind-Blowing Growth Is About to Propel Anthropic Into Its First Profitable Quarter.” Since the beginning of last year, Anthropic’s annual recurring revenue (monthly revenue times 12), has gone from around a billion dollars to around $40 billion. People who know about these things say revenue growth at this rate, and on this scale, is unprecedented—like, in the entire history of capitalism.

And note that, by virtue of Anthropic’s business model, the revenue comes mainly from companies, not consumers—which means that Anthropic AIs are doing a lot of work that human workers have traditionally gotten paid to do. So continued Anthropic revenue growth could mean real impact on the labor force, for better or worse.

And then there’s this: One company where work is being thus automated is Anthropic. So the kinds of AI tools that Anthropic is building accelerate the building of new kinds of AI tools. This kind of positive feedback cycle is what, in theory, could lead to the “singularity”—the point at which change starts happening so fast that God only knows what lies beyond it.

This week at an annual Google conference known as I/O, after the company had outlined plans to sprinkle AI onto pretty much every product it makes, Demis Hassabis, head of Google’s Deep Mind, turned heads by musing, “When we look back at this time, I think we will realize that we were standing in the foothills of the singularity.” On that same day, Anthropic said that Andrej Karpathy, a co-founder of OpenAI and the former head of AI at Tesla, would be joining Anthropic, leading a team that uses Anthropic’s large language model, Claude, to accelerate research on the training of LLMs. In other words: Karpathy’s mission is to accelerate Claude’s self-acceleration.

Well, at least for now there’s a human in that loop. Maybe we should make some progress on this AI regulation issue while that’s still the case.

Subscribe now

Note: Publication of my book on AI, The God Test, is a month away. Maybe you should pre-order a copy now in case the singularity disrupts the world’s electronic payment infrastructure before June 23!

Share

As noted in last week’s Earthling, my view is that the whole world benefited from the Trump-Xi summit. With that in mind, you might think I consider the framing of the survey below—“who benefited more”—unfortunately zero-sum. Well, strictly speaking, no; even in a non-zero-sum game that has a win-win outcome, it’s possible for one side to benefit more than the other. Still, the wording of this question does probably steer some people toward a zero-sum way of thinking about the summit’s outcome. Which I guess means people who replied that Trump and Xi benefited by an “equal amount” deserve credit for nonetheless giving an unambiguously non-zero-sum answer. And—credit where due—MAGA supporters were more likely to do that than any other group.

Share

Banners and graphics by Clark McGillis.

On Thursday the White House issued a social media post that said simply, “American strength back on the world stage.” You might expect such a post to be accompanied by military imagery—maybe an impressive-looking array of soldiers. And this one was. But the soldiers weren’t American. Smartly clad People’s Liberation Army troops, part of the pageantry that had greeted Trump in Beijing, were featured in the opening montage of a video that then showed Trump shaking hands with President Xi Jinping and ended with the two leaders walking side by side on a red carpet.

Apparently the unipolar moment is over. Being one of the world’s two main poles is, these days, the highest geopolitical aspiration of America’s president. “It’s the two great countries,” Trump proudly told Sean Hannity in an interview that aired on Fox News after the US-China summit ended. “I call it the G-2. This is the G-2.”

The US foreign policy establishment was less effusive than Trump about G-2 membership. A Washington Post headline said: “China summit yields Xi’s goal—equal footing with U.S.,” and the subhead added that “the image of peer superpowers during President Trump’s visit displayed a dynamic that analysts say the Chinese have long sought and Americans had resisted.” The analyst quoted by the Post, a Biden administration national security staffer, said that Xi, using “the opulent optics of the visit,” had managed to do something “Chinese leaders have been working toward for decades”—he had made it “clear to the world that China and the United States are the two dominant, equally matched superpowers. There is no going back.”

New York Times reporter David Sanger, the dean of Blob scribes, also depicted the summit as a kind of comedown for America. Whereas Trump was “conciliatory,” Sanger wrote, Xi was “quietly more confrontational,” projecting a “new level of confidence and authority.” In particular: Xi had, according to a Chinese foreign ministry statement, privately warned Trump that “the US must handle the Taiwan issue with utmost caution” if it didn’t want to wind up in a conflict.

Meanwhile, as Xi was drawing red lines, Trump, Sanger noted in a subsequent co-authored piece, was spending his time calling Xi a “great leader” and saying that a garden where the two men strolled contained “the most beautiful roses anyone’s ever seen.” All of this showed, according to Sanger, “how far he has shifted the foundations of American policy toward China in the wake of his humbling retreat from last year’s trade war. He has thrown aside the adversarial approach of his first years in office, the Biden administration, and the beginning of his own second term.”

Works for me! Then again, I’ve always been a sucker for peaceful coexistence. Besides, the non-zero-sum problems that the US and China need to solve are increasingly urgent, and a pleasant stroll in the garden can be conducive to cooperation. In fact, this week the two nations agreed to start a dialogue on AI safety. For now they’re focusing mainly on ways to keep non-state actors from putting AI to malign uses. That agenda will likely grow as the magnitude of the AI issue becomes more evident and the inherently international nature of many AI threats does, too. But for now it’s enough that, as Treasury Secretary Scott Bessent put it, “the two AI superpowers are going to start talking.”

Share

Subscribe now

I think I’d go even further than Sanger in stressing the magnitude of this moment. Trump has “shifted the foundations” not just of “American policy toward China” but of American foreign policy globally. And at the moment, at least, this shift looks auspicious.

Before elaborating, I should emphasize that I’m not calling Trump a visionary. The new and improved foreign policy that I hope is unfolding will, if it unfolds, be one that Trump has more or less stumbled into—and will be different from the one he was pursuing when he started stumbling. Contingency plays a big role in history, and reckless leaders like Trump are particularly good at illustrating that.

Imagine, for example, that the Iran War had gone according to plan: Trump oversees a Venezuela-esque display of military mastery, installs a puppet regime, starts exercising remote control over Iran’s oil spigot, and waltzes into Beijing as king of the world. In that scenario, this summit might have had a different tenor, with Trump demanding more and demanding it loudly, creating the kind of friction he’s so good at creating. And it’s especially easy to imagine that kind of summit if—to add a second power-of-contingency thought experiment—some adviser had last year persuaded him to go easy on the tariffs and other forms of economic warfare, in which case Beijing wouldn’t have been pushed to the point of chastening Trump by playing its rare-earth minerals card.

But those are just thought experiments. In the real world, Trump came to Beijing a humbled man (OK, OK, a closer-to-humbled-than-usual man). What’s more—and what makes me so emphasize the magnitude of this moment—the things that humbled him have implications that go well beyond US-China relations. True, his Middle Eastern demonstration of the limits of American military power has, on the one hand, implications for Taiwan (a fact that has no doubt crossed Xi’s mind). But it also has implications—tectonic implications—for the Middle East itself. As the Beijing summit started, there were reports that Saudi Arabia has proposed a non-aggression pact between Iran and its Arab neighbors. That’s the kind of thing you propose when you realize that your guardian superpower can’t keep you safe—and it could turn out to be the kind of thing that foreshadows the eventual withdrawal of that superpower from the region.

The increasingly clear limits of American military power would exist with or without Trump. They’re a product of various forces, including technological changes that are rendering things like aircraft carriers and tanks less valuable as instruments of aggression. America’s waning ability to intimidate the world with economic weapons also has some deep drivers—such as America’s longstanding and profligate use of economic sanctions, which has hastened “de-dollarization.” (It was Marco Rubio who said three years ago, “We won’t have to talk about sanctions in five years because there will be so many countries transacting in currencies other than the dollar that we won’t have the ability to sanction them.”)

In short: Sooner or later, America was going to have to recognize the limits of its economic and military power and give up on the goal of dominating the world. And I think sooner is better than later. So I guess having a president whose obliviousness to those limits leads him to collide with them in such illuminating fashion is a blessing of sorts.

Share

Unfortunately, the personal qualities that led to those collisions persist. Don’t be surprised if Trump unrolls a regime change operation in Cuba next month—or next week, or in five minutes. And he may renew combat operations in Iran, being the slow learner that he is.

Still, whatever lies ahead, the Beijing summit offers real hope by virtue of its implicit recognition of long-denied structural realities. The age of American hegemony is over. Acknowledging that is a start.

Trump is in one respect well suited to build on that start. He won’t, like Biden and many previous presidents, spend a lot of time and political capital demanding that Xi reform his human rights policies. Trump’s indifference to such concerns may reflect poorly on his character, but it would be a more lamentable trait if the traditional American approach to human rights issues—sermons plus sanctions—weren’t so rarely productive and so often counterproductive, hurting the very people it’s supposed to help.

One thing standing in the way of my dream scenario—a world where the US and China and the world’s other nations cooperate to solve problems they collectively face—is Trump’s aversion to international governance. China under Xi plays a more active role in international institutions than the US under Trump, and Xi talks more about “win-win” outcomes than Trump, who is famously inclined to see life in zero-sum terms.

Still, he emitted fewer zero-sum vibes this week than he often does. Indeed, inherent in the idea of “G-2” is sustained cooperation between the two superpowers. So maybe Trump has more in the way of non-zero-sum impulses than is commonly recognized, and they just need to be nurtured. And maybe it would help if the New York Times and the Washington Post and much of the foreign policy establishment quit giving him negative feedback every time he expresses them.

Share

Subscribe now

Subscribe now

Share

Banners and graphics by Clark McGillis.

Nathan Leamer’s Twitter bio is only ten words long: “Suburban Dad. Work at intersection of tech and public policy.” He’s being too modest. Leamer is executive director of a group called Build American AI, which is backed by the super PAC Leading the Future, which in turn is backed by people like OpenAI President Greg Brockman (to the tune of $25 million) and venture capitalist Marc Andreessen (via his VC firm A16Z) and Palantir co-founder Joe Lonsdale.

Though Leamer seems not inclined to talk about his job, his Twitter feed offers some clues about his duties. One of them, apparently, is depicting influential critics of AI in a negative light. This week Sen. Bernie Sanders hosted a livestream conversation called “The Existential Threat of AI and the Need for International Cooperation.” Joining Sanders were two American AI safety advocates and two AI experts from China—one from Tsinghua University (aka “China’s MIT”) and one from the Beijing Institute of AI Safety and Governance. Leamer tweeted this analysis: “Bernie Sanders decided to let 2 members of CCP-controlled organizations zoom into his anti innovation party at the Capitol.”

Leamer also circulated commentary by other analysts. One tweet he retweeted said that 20,000 people had tuned in “to watch a US Senator cater to the CCP on AI innovation in the US.” Another said: “Incredible to see a US senator endorsing America’s surrender to China.”

I think I sense a theme developing…

Share

To understand why this theme will keep developing—why you’re going to see more critics of AI implicitly or explicitly cast as traitors, and why people like Brockman and Andreessen and Lonsdale underwrite this stigmatization—you have to understand that US-China cooperation isn’t the only, or even the main, thing about Sanders’s AI agenda that bothers them.

Along with Rep. Alexandria Ocasio-Cortez, Sanders has proposed legislation that would pause the construction of AI data centers in the US until Congress enacts AI safeguards. And a cardinal principle of AI accelerationist rhetoric is that any regulation that could slow the development of AI poses a grave threat to America by handicapping it in its presumably existential race with China. Both Americans who joined Sanders in his livestream have been on the NonZero podcast, and one of them, Max Tegmark of MIT, said during our podcast conversation, “Companies in the US have found that by far the easiest way to avoid being regulated is just to say this two-word sentence: ‘But China!’ And then regulators back off.”

So, for Silicon Valley, a machine that smears advocates of regulation as unpatriotic is a dual-use technology: It discourages international agreements that could slow AI down and cut into revenue, and it discourages national regulation that could slow AI down and cut into revenue. Nathan Leamer is, among other things, part of that machine. I hope the job pays well.

And it should! It keeps him busy. When he’s not tweeting, he’s doing things like helping journalists with their research. This week, for example, he was quoted by New York Post columnist Lydia Moynihan in support of her assertion that “AI doomerism—the belief that advanced AI poses an existential threat to humanity—has quietly forged one of the stranger coalitions in modern politics. Socialists, Hollywood unions, Jan. 6 organizers and Chinese government officials are now aligned in trying to slow down American AI development under the guise of saving humanity.”

I don’t want to overdo the distinction between national and international regulation. With this technology, especially, the two are often connected organically. Even if you favor Sanders’s data center moratorium, you have to acknowledge that its effectiveness will be limited by the fact that data centers can be built abroad. Or, to take a proposal that would have a better chance of enactment than a data center moratorium: If you put a heavy tax on data centers, more data centers will be built abroad, so the tax will have slowed things down only modestly and so will buy us only a little more time to ponder and wisely calibrate AI governance. The natural solution would be an international agreement on a minimum data center tax; effective national governance would entail international governance.

This international tax idea isn’t as crazy as it may sound. The Biden administration pushed for an international minimum tax on multinational corporations to stem the “race to the bottom” that encourages countries to lower taxes in hopes of attracting companies. And 136 nations agreed to a 15 percent minimum tax!

The big and fatal obstacle—as is often the case with worthwhile international initiatives—was the US itself. But most of the domestic opposition came on the Republican side, and much of that came from MAGA Republicans—and many MAGA Republicans are alarmed about AI. (Hence that reference to “Jan. 6 organizers” in the New York Post.) On the same day Tegmark appeared on the Bernie Sanders livestream, he appeared on Steve Bannon’s podcast. So maybe it’s not too wild a dream to imagine a left-right coalition that supports not just national but international AI regulation.

Share

Subscribe now

Granted, it’s a pretty wild dream. MAGA Republicans are traditionally averse to global governance in general and cooperation with China in particular. Bannon, when cutting to commercial breaks, sometimes plays a song called “Take Down the CCP.” And I’m pretty sure that if we hold off on the international governance of AI until we see regime change in China, we’ll have waited too long. This week the New York Times ran a story about how helpful consumer-grade AI can be to the aspiring bioweapons maker—and pandemics, like many other bad things AI could foster, don’t respect borders.

Is McCarthyism too strong a word for the tactics of AI industry operatives like Leamer? Well, Leamer is certainly no Joseph McCarthy. He hasn’t, for example, tried to get anybody fired on grounds of ideology. And, Leamer aside, I don’t expect to see anything comparable to the Hollywood blacklists. Fear of China isn’t as intense or widespread as fear of Soviet communism became after World War II.

Still, trying to discredit people with groundless accusations or insinuations of aiding and abetting an enemy (real or imagined) is, among other things, antithetical to the principles of liberal democracy. (The House Un-American Activities Committee, whose investigation of Hollywood in the late 1940s foreshadowed McCarthy’s rise as a red-baiter, certainly earned its name!) Plus: The US tech industry—much to its benefit—includes many people of Chinese ancestry, and they should be able to choose and vocalize their positions in the AI debate without worrying that their ethnicity will make that endeavor particularly perilous. And even when Leamer’s cheap shots, as in this case, aren’t directed at people of Asian ancestry, they still help create an atmosphere in which that peril may be felt.

Few psychological forces are more corrupting of rational debate, or more conducive to oppression and persecution, than the suspicion that members of your tribe are in league with a tribe that’s been deemed an enemy. And this is an especially destructive force to unleash if you’re in the middle of a technological revolution which demands that humankind, for its own good, do a better job of transcending nationalism and other forms of tribalism than it’s ever done before. If I seem hypersensitive to signs of McCarthyism, part of the reason is that I think that’s what we’re in the middle of.

Share

Subscribe now

PS I elaborate on my argument that the AI revolution demands a war on tribalism—and suggest some ways to fight that war—in the closing chapters of my book The God Test, which comes out next month. You can pre-order here if you’re so inclined.

Good news for President Trump! Since he was inaugurated last year there has been an increase in the number of Americans who think US influence in the world has been strengthening! What’s more, the final data point in the graph above comes from late March—nearly a month into the Iran war. So maybe Americans are rejecting the view, widely held abroad, that the war illustrates a declining American ability to use its power effectively. Or maybe, on the other hand, Americans who answered this survey question noted that it didn’t specify whether the influence was for good or ill.

In any event…

The graph below suggests that the war isn’t proving very popular among Americans. Trump’s attack on Iran brought a rally-round-the-flag effect, and so did the attempted attack on Trump last week. But the first of those two effects faded entirely, and the second is showing signs of doing the same.

Subscribe now

Share

Banners and graphics by Clark McGillis.

As the fifth anniversary of my seminal contribution to blobology approaches, there will naturally be widespread clamoring for me to take stock—to deliver a report on the state of the Blob (defined, loosely speaking, as the foreign policy establishment). As I prepare this anniversary opus, I thought I’d give NZN readers a preview—a report on the state of my state of the Blob report.

I’m afraid the news isn’t good.

I’m not referring to the obviously bad news: that one of the key takeaways from Blob studies—about the manifest idiocy of launching regime change wars, especially when the regime poses no imminent threat to any nation and no long-term threat to America—somehow failed to penetrate the brain of the current occupant of the White House, even though as a candidate he explicitly assured us that it had.

No, I bring even worse news than that. Namely: As the age of artificial intelligence dawns, the various people who make a living as Blobsters—the think tankers, the past and present government officials, the military-industrial-complex titans, the influential media figures—show few signs of getting the picture. Whereas the atomic age spawned lots of creative and even enlightened thinking about its revolutionary implications for national and international security, the age of AI seems so far to be having roughly the opposite effect.

In recent days, two paradigmatic examples of this problem have come to my attention. One involves the China Talk podcast, which for better or worse (plot spoiler: worse) has attained some prominence within the Blob, and indeed recently featured two important former government officials sharing their thoughts on artificial intelligence. The other example involves Palantir, the Silicon Valley defense contractor known for managing the AI-infused software that processed the list of opening-day targets for the Iran war—a list that seems to have included an elementary school whose student body is now largely dead.

I always encourage people who don’t know much about Palantir and are curious about it to start with this short video of Palantir CEO Alex Karp speaking at a public event a bit more than a year ago:

Why, when I was a boy, the men who ran the big defense contractors were mild mannered and evinced a sense of decorum—maybe because they perceived that coming off as belligerent and semi-deranged would be bad for their brand. I’d like to think that they understood something Karp doesn’t, but I fear that, actually, we’re now living in an age when coming off as belligerent and semi-deranged can be good for your brand.

Share

I have additional thoughts about Palantir, but they’ll be easier to process if I first spend a little time on China Talk. This podcast’s more dovish detractors sometimes call it China Hawk, and indeed, its host, Jordan Schneider, fits that description. I interviewed Schneider on the NonZero podcast some months ago and tried to challenge his China hawkism. Judging by his recent discussions of AI, my intervention was less than transformative.

Consider, for example, his take on the implications of Mythos, the new but as yet unreleased Anthropic large language model that, as I’ve noted before, has an unprecedented ability to find and exploit software vulnerabilities. Some people think Mythos foreshadows future AI threats to international stability that are best addressed through international cooperation. Nvidia CEO Jensen Huang recently suggested as much, a suggestion that triggered this response from Schneider on his podcast:

[Jensen] said, “Look, this is something that the US and China need to solve with dialogue.” I am sorry—No one is going to solve the fact that states want to hack other states and other state systems and read their secrets, and, like, maybe make things blow up, through dialogue. I mean this is not something like a global pandemic where it sort of threatens everyone.

Actually, this is like a global pandemic. Powerful and autonomous AI hackers, if not kept under control, could be very bad for the US and China—and lots of other nations. For example: (1) These AIs could fall into the hands of a non-state actor, and the damage done by self-replicating autonomous hackers could cross borders, maybe resulting in the destruction of big chunks of the planet’s communications or energy or financial infrastructures; (2) One of the two AI superpowers, mindful of the growing cyberpower of the other, could misread some cyber-snafu as the opening wave of a cyberattack and then counterattack in fearful desperation, starting a conflict that was catastrophic for both.

In both scenarios—and others I could dream up—the key thing is that a lose-lose outcome is possible, so the game is non-zero-sum. In other words: It’s like a pandemic—and like various other phenomena that should in theory lead nations to cooperate out of self-interest.

This isn’t rocket science! It’s game theory 101—and back when the foundations of nuclear strategy were being laid, awareness of these game theoretical dynamics pervaded the discourse. Maybe that explains why the calls for US-China cooperation on the issues raised by Mythos have come mainly from old timers—like New York Times columnist Tom Friedman and the venerable Economist magazine and, well, me.

The rising generation of Blobsters, in contrast, seems curiously uninterested in even discussing, much less pursuing, such possibilities. A good illustration is a different recent episode of China Talk, featuring two Blobsters of some stature: Ben Buchanan, special advisor for AI in the Biden administration, and Michael Sulmeyer, assistant secretary of defense for cyber policy in the same administration. Though the entire conversation was about the implications of Mythos, neither mentioned the possibility of any kind of international agreement on AI cyberweapons—no treaty, no protocol, no memorandum of understanding, no nothing.

There was brief discussion of international “norms” that might constrain certain uses of AI. Buchanan suggested that, once AI gets better at helping people make bioweapons, then “maybe the norms save us at that point” but “I kind of doubt it, to be honest.” Sulmeyer sounded equally fatalistic. “Have we already missed that moment in AI? I’m not saying that a normative regime, you know, that we should be spending a ton of effort on that or not, just that if you’re asking about norms, that would be, I guess, my question: Is it already a little too late?”

Share

Subscribe now

But there’s another path that Buchanan and Sulmeyer are sure it’s not too late to pursue. “Near and dear to my heart is building an American lead in AI,” said Buchanan—and the conversation about Mythos had “reaffirmed the importance of doing that.” Sulmeyer echoed the importance of “the leaders of America’s warfighters” understanding how important it is that “our offensive cyberoperators maintain and extend a competitive advantage.”

Meanwhile, Chris McGuire, their colleague in the Biden administration and now a senior fellow at the Council on Foreign Relations, wrote in the Financial Times: “The consequences of losing the AI arms race are no longer theoretical. AI models are now the decisive offensive and defensive tools in cyber space, and American and allied cyber security depends on maximizing the US lead over China in AI.” McGuire advocated tightening already tough restrictions on the export of chips and chipmaking equipment to China.

So what could the success of McGuire’s plan look like? Here’s a possibility you won’t hear from him: If America’s estimated seven-month lead in the AI race persists or even grows, and AI approaches levels of power that could confer utter dominance on the nation in the lead, the incentive of China to stage a pre-emptive strike would grow (as noted in a much discussed paper called “Superintelligence Strategy,” co-authored by Dan Hendryks, with whom I discussed it on the NonZero podcast). For example, China might attack Taiwan, the location of the factories that produce all those advanced AI chips that, thanks to US export controls, flow to the West but not to China.

It strikes some people as almost unbearably ironic to say that America’s national security could be served by restraint in the development of powerful weapons. But life is full of irony! And there was a time when influential people in the national security establishment understood that. More than half a century ago, some of them realized that, though “missile defense” sounds like a security blanket, it could be the opposite: If nuclear powers started developing missile defenses on a large scale, that could increase the chances of nuclear war by undermining the logic of mutually assured destruction. So Richard Nixon—no hippie peacenik—signed the Anti-Ballistic Missile Treaty; he chose to keep America’s nuclear missiles vulnerable because, so long as the Soviet Union verifiably reciprocated, that would keep both nations safer.

I grant that maintaining global stability in the age of AI is much more challenging than it was in Nixon’s day. To take just one example: A key to nuclear deterrence is that you’d know where incoming missiles were fired from, so the attacker could expect retaliation—and, as many analysts have noted, cyberattacks, in contrast, don’t generally have a clear return address. So, in the age heralded by Mythos, you can’t count on deterrence to keep the peace.

But don’t bother Alex Karp with details like that. Palantir this week posted a long tweet that was a bullet point summary of the bestselling book The Technological Republic, which Karp co-authored. Bullet Point number 12: “One age of deterrence, the atomic age, is ending, and a new era of deterrence built on AI is set to begin.”

It will definitely be a new era—we agree on that.

Feel free to pre-order my book on AI, which comes out in June:

Subscribe now

Share

Banners and graphics by Clark McGillis. “The Blob” lede art by Nikita Petrov.

Here’s the link for that April 25, noon ET Zoom call with paid subscribers:

Read more

Save the date: I'll be hosting a Zoom discussion for NonZero members next Saturday, April 25th, at noon US Eastern Time. The main topic will be AI, though some discussion of the Iran war (or, God willing, the war’s end) will probably be in order. A link to the Zoom call is at the bottom of this newsletter.
—Bob

The knives are out for OpenAI CEO Sam Altman, both figuratively and pretty close to literally. Last week—a few days after publication of a New Yorker piece featuring tons of anonymous attestations to Altman’s duplicitousness, and a few days before a Wall Street Journal piece about his conflicts of financial interest—an anti-AI extremist threw a Molotov cocktail at Altman’s house.

Altman suggested a causal link between the two kinds of attacks. In a blog post about the attempted firebombing, he called the New Yorker article “incendiary” and noted that it came in a time of “great anxiety about AI.” Maybe, he mused, the attack on his home was testament to “the power of words and narratives”.

A debate about the relationship between strong words and radical action was already happening in AI circles, owing to an armed attack a week earlier on the home of an Indianapolis politician who supported the construction of a local data center. The Altman attack turned the volume up a notch.

Nirit Weiss-Blatt, who writes an anti-AI-alarmism newsletter, assessed some Discord posts written by the accused firebomber and concluded that he had undergone “a long radicalization process shaped by ‘AI existential risk’ rhetoric.” OpenAI’s global policy chief, Chris Lehane, connected the Altman attack to the “very, very negative and dark view of humanity” held by “the Doomers.” Lehane said, “When you put some of those thoughts and ideas out there, they do have consequences.” Putting a finer point on it, White House AI adviser (and wealthy veteran of Silicon Valley) Sriram Krishnan highlighted the title of a book co-authored by Nate Soares and Doomer-in-chief Eliezer Yudkowsky. “This is the logical outcome of ‘If we build it everyone dies,’” Krishnan tweeted.

Here’s my radical take: If there’s a sense in which finger pointing is in order, some fingers should be pointed toward Silicon Valley—pointed at the culture of self-serving tech libertarianism that people like Krishnan represent. As exhibit A, I direct your attention to last week’s edition of the very influential podcast All In, whose cast of tech bro regulars includes White House AI Czar David Sacks and which is to Silicon Valley roughly what Pravda was to the Soviet Union (and which, for that matter, is to the Trump administration roughly what Pravda was to the Soviet Union).

Share

Subscribe now

But first a note about the senses in which finger pointing is and isn’t in order:

Obviously, none of us—not the New Yorker, not Krishnan, not Sacks, not Yudkowsky, and, above all, not me—is morally responsible for the actions of people who are inspired or provoked by our words to do things we don’t espouse and didn’t mean to encourage. And if we’re held legally responsible for such things, that’s a threat to the foundations of liberal democracy.

That said, it is legitimate, and can be healthy, to argue that certain kinds of speech, though constitutionally protected, and perhaps morally defensible as well, do make certain kinds of reprehensible behavior more likely. Among other virtues, getting clearer on the causal connections between speech and bad behavior can help conscientious people refine their messaging to reduce its cost/benefit ratio. This clarity can also inform people’s judgments about which kinds of speech, though constitutionally protected, nonetheless deserve moral condemnation.

Now back to All In. Last week’s episode began with a discussion of Anthropic’s new but unreleased large language model Mythos, which I wrote about in last week’s Earthling. Mythos, apparently, has demonstrated an uncanny ability to find, and in many cases exploit, vulnerabilities in software; it has found holes in all major operating systems and browsers, among other platforms. So Anthropic is sharing Mythos with companies that maintain big parts of the digital infrastructure, like Microsoft and Apple and Cisco and Google, so they can find holes and patch them.

The All In discussion of Mythos began with guest Brad Gerstner, a Silicon Valley investor, who applauded Anthropic’s actions and used them as an object lesson in how little regulation AI needs. “What I like about this is they didn’t need government to hold their hand on this,” he said of the team at Anthropic. “We have plenty of government regulations.”

Now let’s think about this.

If what Anthropic says about Mythos is true, then we may have just dodged a very big bullet. This is an AI that, in the wrong hands, could do God knows what—take down power grids, vacuum up the life savings of retirees, induce a panic or a stock market crash by hijacking social media accounts en masse. And Gerstner seems to agree that the stakes are this high. If Mythos had been released, he said, it “would wreak havoc”—it would have “broken a lot of core things on the internet” and “allow offensive hacking.” Yet he’s happy to leave the decision about releasing such tools in the hands of whoever happens to be running whichever AI company happens to develop the next comparably dangerous tool—and the one after that and the one after that and…

Share

And it’s not like envisioning an occasionally reckless CEO of less than sterling character takes a wild leap of imagination. That New Yorker piece quoted one acquaintance of Sam Altman calling him a “sociopath” and a former member of OpenAI’s board describing him as having an “almost a sociopathic lack of concern” for the consequences of his deceptions.

The source of Gerstner’s faith in the judgment and character of all present and future AI CEOs is a little unclear. At one point he seems to suggest that self-interest will lead them to do the right thing. What Anthropic did was in “the best long-term interest of the company,” he says. Yet he prefaces his remarks by saying Anthropic deserves “a ton of credit here” and ends them by saying that Anthropic CEO Dario Amodei and his team “deserve a lot of credit”—even though credit isn’t something we normally award for pursuing self-interest.

Maybe Gerstner’s point was that pursuing your long-term self-interest can be hard when you’re in a fiercely competitive AI race, and releasing a world-beatingly powerful model could generate a lot of revenue and steal the thunder of your most bitter rival. If so, I agree. Which is exactly why I say we shouldn’t leave these things entirely in the hands of AI companies. Indeed, given the rumors that OpenAI will soon announce a Mythos-level model, I don’t feel wholly confident that either Amodei or Altman will in the coming months exercise as much self restraint as society deserves.

By the way, the investment company Gerstner runs, Altimeter Capital, has stakes in Anthropic and OpenAI worth billions of dollars. So it’s possible that his views on government regulation of the AI industry aren’t entirely free of bias.

All In host Jason Calakanis did disclose Gerstner’s stake in these companies, so it’s not like this was a covert influence op. But in a way that’s my point: I think one thing that drives some doomers crazy—figuratively and perhaps in the case of Altman’s assailant literally—is the perception that serious discussion about regulating AI is being thwarted by rich and powerful people who are serving their own interests at the expense of the public interest. These Silicon Valley potentates wield great influence via various levers of power, and they do so conspicuously and unabashedly.

Subscribe now

Share

For example, Greg Brockman, president of OpenAI, has donated $50 million to Leading the Future, a Super PAC that’s devoted to squashing various regulatory initiatives and kneecapping congressional candidates who support them. I assume he’s happy with his work so far; as of now there is zero meaningful federal regulation of AI. And, lest any states fill the void, Leading the Future backed a law that would have banned all AI regulation at the state level for 10 years.

I’m not saying Brockman bears any moral responsibility for the firebombing of Sam Altman’s house—any more than Eliezer Yudkowsky does. But if you’re speculating about the causal forces that came together in the firebomber’s presumably unbalanced mind to catalyze his crime, I don’t see any reason to give more weight to Yudkowsky’s arguments than to the perception that powerful people like Brockman are making it hard for such arguments to get a fair hearing via conventional political channels.

On this week’s episode of the New York Times tech podcast Hard Fork, co-host Casey Newton, in the course of discussing the attack on Altman’s house, said the following:

This AI moment that we’re living through is a top-down moment. It did not rise up from the grass roots from a bunch of nerds getting together in their garages and training frontier models. It was a small group of really smart people who were able to get access to massive amounts of capital from the elites in our society, and they’re now mounting this effort to build it very quickly, deploy it very quickly without a lot of guard rails. I think when the average person looks at this, they think, not only did I not ask for this but I have no meaningful control over it. And I think that’s a big reason you’re seeing people so furious—because I think, particularly on the left, this just looks like mostly a right wing elite project that’s being championed by President Trump and the many venture capitalists that are in his administration, and if you’re already worried it’s going to take your job, and you think you don’t have any control over it, well of course you’re going to hate it.

That sounds about right to me. People worried about the impact of AI look around and see no serious consideration of the possibility that this technology is moving too fast—at least, no consideration that has had any impact at the policy level. And what they do see—on the All In podcast and elsewhere—is rich, powerful people who make facile and obviously self-serving arguments yet carry the day. And often these arguments are aired in forums that, like the All In podcast and God knows how many pricey tech conferences, have the trappings of debate but lack the diversity of viewpoint that would make the debate meaningful.

Share

A question that may have more relevance to all this than meets the eye: Why did we see, in the course of a couple of weeks, both the New Yorker’s massively documented piece about Sam Altman’s deceptive tendencies and the Wall Street Journal’s piece about his conflicts of financial interest? After all, the outlines of both stories have long been evident. Karen Hao’s book Empire of AI, for example, convincingly depicted Altman as a slippery character whose word isn’t exactly his bond.

Maybe part of the answer is that, as OpenAI has over the past half-year lost ground to Anthropic, and has seemed less and less sure footed, discontent with Altman’s performance as CEO has grown. And the less confident OpenAI investors and workers feel about the future value of their equity in the company, the more open they are to change in the CEO suite—so the more willing they are to share with journalists criticisms of the current CEO that they’d previously kept under wraps. Indeed, the Wall Street Journal piece reports that there is more and more talk, among OpenAI stakeholders, about replacing Altman.

This is just speculation on my part, and it could well be wrong. But, even so, the timing of this sudden proliferation of doubts about Altman’s character is consistent with a view held by some critics of the tech industry: that Silicon Valley is fine with CEOs who lack integrity so long as they’re good at making money. This is one of many unflattering perceptions that it may not be in Silicon Valley’s long-term interest to foster.

Share

Subscribe now

Share

Banners and graphics by Clark McGillis.

And, finally: Here’s the link for that April 25, noon ET Zoom call with paid subscribers:

Read more

Save the date: I'll be hosting a Zoom discussion for NonZero members on Saturday, April 25th, at noon US Eastern Time. The subject will be AI and, to some extent, war—and the particular connection between the two that’s laid out in the piece below. A link to the Zoom call is at the bottom of this newsletter.
—Bob

“The week that changed everything” is a phrase that shouldn’t be tossed around lightly. After all, “everything” covers a lot!

Also: In what may be the phrase’s most common context—a Christian context—it refers to the week that brought the Crucifixion and Resurrection of the Messiah and hence the promise of salvation for humankind. And that’s a high bar.

So I’m going to hold off for now on calling this past week the week that changed everything. But I’m going to argue that, depending on how things play out, a title this exalted could wind up being in order. This was a week that saw seismic tremors at two different levels—one geopolitical (in the Middle East) and one technological (in artificial intelligence). And there was a connection between the two that isn’t yet visible but could prove momentous. The connection could even prove salvific—salvific as in salvation, as in the salvation of humankind. And I’m not kidding, so long as “salvation” refers not to an afterlife or the cleansing of our sins but to saving humankind from various apocalyptic outcomes, up to and including extinction.

It’s possible that I’m reading too much into this. I have a book coming out in June called The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning. And when you have a book coming out you tend to see the world through that lens, and this can lead to distortion. You can, for example, see corroboration of your book’s thesis everywhere you look. And such corroboration is what I see when I look at this week’s news. That’s not necessarily a cheering thought, because my book’s thesis isn’t wholly upbeat. Still, I do see, in this week’s news, some cause for hope—reason to think that maybe the dark scenarios laid out in my book won’t come to pass and the brighter scenarios laid out in it will.

But, again, I could be wrong; AIs aren’t the only beings capable of hallucination. You be the judge.

One of the book’s central points is that if we’re going to successfully navigate the AI revolution—avoid traumas and catastrophes that range from social chaos to planet-wide authoritarian rule to nuclear war to complete annihilation—we’ll have to cross the threshold to true global community. The world’s nations have to confront this challenge collectively—build new international rules and norms—or else watch in dismay and intermittent terror as a technology that accelerates without constraint or guidance strips us of agency. (If this sounds too dramatic, please suspend judgment. I devote a fair amount of the book to laying out the evidence and logic behind my more dramatic-sounding claims.)

Share

Subscribe now

Much of what we need to do in controlling AI comes under the rubric “international governance,” but I think this phrase, radical though it sounds to some people, runs the risk of understating the challenge. After all, we have international governance now. Various treaties, such as the Nuclear Non-Proliferation Treaty and the Law of the Sea Treaty, qualify for that label. And—don’t get me wrong!—these treaties are nothing to sneeze at. But few of them are global, and they often lack enforcement mechanisms, and compliance can be spotty, and they have a tendency to fray. I think AI will ultimately call for structures of international governance more ambitious and nuanced than the kind we’ve had. And, as if that weren’t a tall enough order, these structures will have to be bolstered by new norms, and these norms will have to get the kind of respect that a society’s most important norms get.

This is why I used the phrase “true global community” above. I don’t think our species will be able to muster the focus and commitment this mission demands unless we can sustain at least the minimal degree of harmony that the term “community” implies. That doesn’t mean kumbaya-level harmony, but it does mean we can’t keep getting bogged down in the kinds of wars that have plagued humanity since (at least) the dawn of history.

This points to one reason I found this week’s Iran War ceasefire, partial and tenuous though it is, heartening. I’m hoping it can usher in not just peace, but a peace that has a particular kind of psychological impact—a peace that highlights the frequent futility and absurdity of war. After all: The two aggressors, the US and Israel, spent God knows how much money and killed God knows how many innocent people, and not only failed to reach their stated war aims but may have handed their enemy a new and potent form of regional influence: ongoing control of the Strait of Hormuz. (I elaborate on some of this in a video op-ed I posted Friday about my hopes for the ceasefire—and argue that this new Iranian influence, though anathema to Israel and America, could be a stabilizing and pacifying force in the near term.)

What’s more, this war has been nothing short of a political debacle for the leader of the more powerful of the two aggressor nations, Donald Trump. So the perpetrator of an egregious violation of the UN Charter got powerfully negative feedback. That’s the kind of feedback we need if we’re going to restore some respect for international rules and norms, a pre-requisite for building a true global community.

As if Trump’s humiliation weren’t enough cause for celebration, the transparency of his motivations helped highlight the absurdity of the political dynamics that so often lead to war: National leaders inflate threats abroad to foment fear at home and thus benefit politically by posing as guardians of national security. Both Trump and Benjamin Netanyahu are veteran practitioners of this craft, but in this case Trump practiced it so ineptly as to invite widespread ridicule. He literally couldn’t think of a single quasi-plausible threat that Iran posed to the US, even though he deemed the Iranian menace so grave as to warrant an assault that killed thousands of people, wiped out untold infrastructure, and destroyed numerous cultural treasures.

Trump’s crudeness—the playground taunts and macho chest-pounding—can get tiresome, but it does have the virtue of clearly demonstrating what drives him and many other warmongering politicians: They’re just alpha males trying to stay on top of the heap. It would be hard for him to more vividly convey the primitive and visceral energies that fuel wars without actually putting on an ape costume.

Share

I don’t want to get carried away. We are not on the verge of ending war forever, and in fact it’s not even clear that we’ll succeed in ending this war. Still, this war had some of the hallmarks of a landmark. In The God Test I write, “Seeing the absurdity of the average conflict doesn’t make any particular conflict easy to resolve, and it doesn’t mean there’s never one side in a conflict that deserves support. Still, if the amount of change we need on this planet is as great as I think it is, it may be necessary for large numbers of people to look around and say, ‘Folks, this is ridiculous. We can’t go on like this.’ ” What we’ve seen over the past six weeks is a war that, more than any American war I’ve lived through, invites that kind of reaction. Folks, this was ridiculous. It was almost a caricature of the folly of war, and I hope it will be seen that way more and more as it recedes into the past.

So that’s the seismic activity at the geopolitical level that I alluded to above—a war that seized the world’s attention by imperiling the global economy and that, however unwelcome in itself, could have a welcome impact on the perception of war itself, an impact that I hope is the beginning of truly tectonic change, of a sustained drift away from war.

Of course, a hope is just a hope. One conspicuously absurd war won’t by itself transform the world’s view of war, much less build a global community. That construction project will require additional impetus. Which leads us to the other level at which seismic change was evident this week: the technological level, where that very impetus began to materialize.

This week Anthropic announced that it has a new large language model that’s way more powerful than past models—so powerful that it poses a threat to the world’s information infrastructure. This model can find otherwise unfindable vulnerabilities in software—and in fact, said Anthropic, has found previously unknown holes in every major operating system and browser. And in most cases, it is able to then build a tool to exploit those vulnerabilities. So if this model fell into the wrong hands, that could be big trouble—which is why, says Anthropic, there are no plans to release the model for the time being.

Some people have wondered whether Anthropic’s claims about the terrifying power of this new model are mainly marketing hype—a suspicion that isn’t exactly discouraged by the model’s name: Mythos. But Anthropic is making Mythos available to big companies that maintain important parts of the digital infrastructure, like Microsoft, Apple, and Cisco. So if these claims were hugely exaggerated, word of that would get out—especially since one of the companies, Google, is a direct competitor of Anthropic’s and would presumably prefer not to admit that Mythos is significantly more powerful than its own frontier model, Gemini, in such an important domain. (Mythos, says Anthropic, wasn’t designed to facilitate cyberattacks—this capability was just a byproduct of designing it to write computer code and handle intellectual challenges more generally.)

AI, as Mythos demonstrates, is a good example of a technology that can’t be adequately governed at the national level alone. Today it’s America that has the most potent AI hacker, but tomorrow it could be China. In fact, we can’t be quite sure that China doesn’t have a potent AI hacker now, because the two countries are making no serious effort to collaboratively address the emerging challenge they face and so aren’t doing a lot of communicating on this subject.

That gives me an uneasy feeling. As I write in The God Test:

These kinds of dangers—AI-abetted biological virus, AI-abetted computer virus, AI-infused cyberweapon, rampantly destructive AI agent—and various others make it harder for any nation to feel safe unless it has some confidence that things are under control in other nations. And it’s hard to get that kind of confidence without international agreements that qualify, in at least some sense, for the term “international governance.”

Share

Subscribe now

I’m happy to report that the need for international coordination on AI is appreciated by some influential people. New York Times columnist Tom Friedman, assessing the Anthropic news, wrote this week:

The solution—this may shock people—must begin with the two AI superpowers, the US and China. It is now urgent that they learn to collaborate to prevent bad actors from gaining access to this next level of cyber capability. Such a powerful tool would threaten them both, leaving them exposed to criminal actors inside their countries and terrorist groups and other adversaries outside. It could easily become a greater threat to each country than the two countries are to each other.

But other observers have reacted in roughly the opposite fashion, saying there’s no realistic hope of deep trans-Pacific cooperation, so America must double down on its AI race with China. This will no doubt be a common reaction.

Another common reaction, I suspect, will be that we should immediately set about deploying Mythos as a weapon against China. This week the well known social media influencer Noah Smith, in reference to a story about recent data theft from a state-run Chinese computer, tweeted: “Just today, someone asked me: If our government knew about Mythos ahead of time, why didn’t it team up with Anthropic to hack China before announcing the model’s capabilities to the public?”

If our government hasn’t done that, the reason may be the Trump administration’s ongoing spat with Anthropic. But you can bet that people in the administration are pondering the question Smith raised. And you can bet that people in Beijing are pondering it. And you can pretty much bet that nobody in the Trump administration is wise enough to ponder the implications of Beijing pondering it. Here’s one such implication: When nations get insecure about the emerging capabilities of another nation, they sometimes take dramatic action. Just ask Bibi Netanyahu. And when the emerging capability is ultra-powerful hacking that, for all you know, is already being deployed against you, a nation can get insecure and reactive, and interpret ambiguous evidence as attack and respond accordingly.

The principle this points to—the potentially stabilizing effects of mutual transparency between the US and China—is just one of many AI-related issues that deserve lots of attention and aren’t getting much. The same might be said of this week’s Anthropic news itself. (The Times, notwithstanding Friedman’s column, ran its news story about Mythos in the business section—and, in the physical edition, below the fold.) But this story is still filtering through tech podcasts and other niche media and is destined to get more mainstream attention for some time to come.

I’m hoping that, as more and more people process the story, they’ll see its significance. I’m also hoping that they’ll understand one reason its significance took a while to sink in: Because news about it was obscured by news of war. To put a finer point on it: Tuesday—the day Anthropic released the news—was also the day Trump declared that if Iran didn’t bend to his will, “a whole civilization will die tonight.”

It’s understandable that this bizarre and vile and ultimately ephemeral threat to a great civilization absorbed our attention while news of a more deeply rooted and enduring threat to civilization writ large flew under the radar. Still, folks, this is ridiculous—that we’re letting primitive and malicious creatures like Donald Trump and Bibi Netanyahu dominate the stage while issues of critical importance to the whole planet languish in relative obscurity.

Maybe reflecting on the absurdity of Tuesday’s allocation of attention will help us eventually get our priorities straight. So remember the date: April 7, 2026. We can call it Planet of the Apes day.

Share

Subscribe now

PS: I taped a podcast conversation this week with Shakeel Hakim, editor of Transformer, a newsletter about AI. We taped the conversation Tuesday, right before the Mythos news broke (or at least before it reached my news feed). But we did agree on some things relevant to Mythos: (1) AI demands international governance; (2) It may take a catastrophe to focus the world’s attention on that fact—but, as Shakeel pointed out, we can hope that this service will instead be performed by a “near miss,” a narrowly averted catastrophe. Well, so far, at least, Mythos hasn’t brought a catastrophe and so is the closest thing we’ve had to a “near miss.” Let’s not waste it.

PPS: I notice that the Amazon page for my book offers a “pre-order price guarantee”—an opportunity to lock in the current price just in case the price changes between now and June. Now, TBH, the chances of the price rising over the next two months are, to judge by past publishing experience, roughly zero. OTOH, the price of oil hasn’t returned to its pre-war level, and I’ve never seen an Amazon delivery truck that had solar panels, so you never know…

Share

Note: Those final data points are from before the Iran War.

Subscribe now

Share

Banners and graphics by Clark McGillis.

And, finally: Here’s the link for that April 25, noon ET Zoom call with paid subscribers:

Read more

Why is the current war happening? If you want to answer that question in a broad sense—in a way that applies not just to the Iran war but to other needless bursts of carnage of the past and future—I would direct your attention to an exchange that took place this week on a New York Times podcast called The Opinions.

The exchange was between Times columnist David French and retired four-star General Stanley McChrystal, and it was an archetypal interaction between two kinds of mindsets—the kind that keeps getting us into wars and the kind that, if it were more common, could keep us out of them.

The roles played by the two men aren’t what you might expect based on their job descriptions. It wasn’t the career Army officer who exemplified the narrowly tribalistic perspective and the writer for the liberal media who offered the more balanced and pacific view. Rather, it was the professional soldier who brought the enlightenment and the journalist who lacked it—and who showed no signs of absorbing any of it.

In a way this isn’t surprising. What McChrystal crucially evinced is the ability to see things from the point of view of other people, including adversaries and enemies—an ability any good military strategist needs. In fact, though I typically call this skill “cognitive empathy,” it is, when employed in certain contexts, also called “strategic empathy.”

As regular NonZero readers know, I consider cognitive empathy a kind of untapped superpower—a human potential that, if more widely realized, could work wonders. For example: It could transform American foreign policy from the blunt and brutal instrument it’s become into something that’s actually good for the world.

A second reason the roles played by these two men isn’t surprising is that French first got wide public exposure in 2016 when Bill Kristol proposed that he run for president. And if Bill Kristol—who as a leading neoconservative played a non-trivial role in getting the US into the Iraq War—would like to install you in the Oval Office, it’s a safe bet that you’re no slouch in the mindless tribalism department.

French was an officer in the Army Reserve, and, like McChrystal, served in the Iraq War. He begins their conversation by noting this shared history and using it to explain to listeners what he calls “the veteran’s perspective on this conflict” with Iran—a perspective he clearly is assuming McChrystal shares with him. This perspective, he says, is “different than the perspective that a lot of the folks who didn’t serve, especially in Iraq, have” and is grounded in awareness of “Iran’s role in the Iraq war, and the losses it inflicted upon us.” He recalls that, while he was in Iraq, “we lost guys to explosively formed penetrators planted by Iranian-backed militias” and then invites McChrystal to take it from there—to elaborate on “what has been the recent American experience in our long-running conflict with Iran.”

McChrystal, though, wants to take the story back in time, not forward. He says, “If we go back to the American experience, starting in 1979,” we remember that “the American embassy [in Tehran] was seized, and there were people chanting ‘death to America’.” And “that was very upsetting,” especially since, before the revolution of 1979, Iran had been America’s staunch ally. So, says McChrystal, post-revolutionary Iran seemed like “a recalcitrant enemy that hated us for some reason we couldn’t really understand.” And that attitude toward Iran was reinforced by the Iraq war experience French had just described. In Iraq, said McChrystal, “they were killing us and we were killing them,” and Iran increasingly seemed like “our lifelong enemies.”

But, he then adds, “I think that’s only part of the story.”

At this point French is either oblivious to where McChrystal wants to take the story or determined not to let him go there. Whereas McChrystal wants to talk about the roots of Iranian hostility toward America, French just wants to keep documenting the hostility. He says, “Well, if you’re going to say that’s only part of the story, we have to keep going.” He notes that, though the “surge” of US forces seemed to have smothered the Iraqi insurgency by 2009, “the story doesn’t end there, of course, General. The story keeps going, and Iranian-supported militias have been a thorn in our side, really, in Iraq ever since.”

French now invites McChrystal to sustain this theme: “Let’s pick it up after the surge. What happens next?”

McChrystal, God bless him, replies, “Well, let’s really pick it up before that, because I think it’s important… It really starts in 1953, when the US and British intelligence services overthrew the constitutionally elected prime minister” of Iran and installed the Shah, who “oppressed the people tremendously, particularly through SAVAK, the secret police. So when the Iranian revolution happens, we may be surprised [but] the Iranian people are not surprised. And when they suddenly say ‘Death to America,’ most Americans will say, ‘What’s your problem? Why are you angry at us?’ ”

Share

Subscribe now

McChrystal’s narrative continues: In 1980, Iraq invaded Iran, starting an eight-year-long “brutal bloodletting” (during which the US supported Iraq with intelligence and other resources). And in 2002, McChrystal notes, George W. Bush designated Iran and Iraq as the two Middle Eastern members of the “axis of evil”—something that didn’t exactly incline Iran to roll out the welcome wagon when Bush invaded Iraq the next year and American troops were suddenly occupying Iran’s next door neighbor. Hence the Iran-backed Shia militias in Iraq that, in French’s telling, are where the story begins.

There are lots of details that McChrystal skips that would have bolstered his narrative—like the fact that, only a few months before that Bush “axis of evil” speech, in the wake of 9/11, Iran had collaborated with the US to pave the way for the American invasion of Afghanistan. (Fun fact: Qasem Soleimani, the Iranian military commander who in 2020 was assassinated by Trump via missile strike, was in 2001 on the ground in Afghanistan, using his connections in the country to help smooth the arrival of American troops. Imagine Soleimani’s surprise when he learned that, after his tour in Afghanistan, he would be reassigned to the axis of evil.) Still, McChrystal fleshes the history out enough so that his basic point is clear. As he puts it: “If we don’t understand that journey to this point, we don’t understand the attitudes that are going to drive decisions people make.”

But French, in a subtle but important sense, persists in not getting the point. He says, “I’m so glad we have dived into this from the Iranian perspective, because I think understanding the Iranian perspective really helps us maybe understand how the rest of this war might go. What kind of staying power, for example, the Iranians might have.” Elaborating, he says that “the level of commitment that exists within the regime is just something that’s theological.” Sometimes it’s even “apocalyptic,” he adds. He then relays an anecdote he heard in Iraq about how fanatical some of those Iran-backed militiamen were.

For all I know, McChrystal would agree with this emphasis on theology and apocalypticism, but it wasn’t implied by anything he’d just said, and in fact it seems closer to the opposite of his upshot. McChrystal seemed to be saying, among other things, that if you understand Iran’s history, you shouldn’t be surprised by Iranian wariness of, mistrust of, and hostility toward the US. These things don’t need some kind of special explanation grounded in Islamic theology or fundamentalist apocalypticism or Persian philosophy or anything else.

French’s depiction of the Iranian regime as intensely religious in its motivation is a time-honored trope of neoconservatives and other hard-core Iran hawks. Hence the boilerplate references—in the writing of Bret Stephens in the New York Times, Walter Russell Meade in the Wall Street Journal, David Frum in the Atlantic, and so on—to “the mullahs” who run Iran. So long as these writers can keep us thinking of Iran’s leadership as zealously religious, they’ll have an easier time convincing us that we shouldn’t let Iran do things every other country is allowed to do—like enrich uranium to a low level for (verifiably) civilian use or have an arsenal of ballistic missiles with conventional warheads.

Share

Subscribe now

The threat that McChrystal’s world view poses to Iran hawks was illustrated this week on another New York Times podcast—Ross Douthat’s Interesting Times. Douthat’s guest was Mark Dubowitz, CEO of The Foundation for Defense of Democracies, a deeply pro-Israel and anti-Iran think tank. After Douthat suggested that the US-Israeli strategy of “decapitation” via the serial assassination of Iranian officials naturally leads Iran to escalate, Dubowitz countered, “Well, my point is that they have been escalating against us and against Israel and against their neighbors for decades… They continue to go up the escalation curve as they’re prepared to take more and more risk against us.”

In other words: The side of the story that a broader, McChrystalite perspective includes doesn’t actually exist; since the dawn of time, Iranian “escalation” has been intrinsically driven, unaffected by exogenous variables. Dubowitz makes no mention of the time in 2024 that Israel, out of the blue, lethally bombed an Iranian consulate in Syria, or the time, in 2025, when Israel, out of the blue, staged a massive air assault on Iran, or the time a month ago when Israel and America, out of the blue, staged another such assault, this time, for good measure, killing Iran’s leader and several family members. So, needless to say, Dubowitz also fails to mention that the only three times in history Iran has started firing ballistic missiles at Israel—the missiles that Dubowitz and Stephens and Meade and Frum says Iran’s “mullahs” shouldn’t be allowed to have—were in response to those three unequivocal provocations.

If Dubowitz were to acknowledge such facts, he might have to entertain the possibility that there’s some truth to the claim that Iran’s ballistic missiles are meant to serve a defensive function—as a deterrent to Israeli attack. And if Dubowitz headed down that path, who knows where he might wind up? He might even start entertaining the possibility that, whatever the intrinsic motivations of such Iranian “proxies” as Hamas and Hezbollah, they, too, from Iran’s point of view, are meant to serve a deterrent function. (It was only after Israel degraded both to the point of near-impotence that it started launching military assaults on Iran—so apparently they had indeed been serving that function.)

And this, in turn, could lead Dubowitz to what for him would be a kind of hell on Earth: a vantage point from which both Israel and Iran are seen as countries that feel deeply insecure and do things out of defensive motivation that look offensive to the other side. In this view of the Israel-Iran relationship, there is a basically symmetrical cognitive empathy deficit, and its malign effects intensify via a positive feedback cycle, and to the extent that either side is to blame, both sides are. And if there’s one thing the Mark Dubowitzes of the world hate, it’s moral symmetry.

President Trump seems to have accepted the one-sided narrative that America’s many influential Iran hawks have promulgated. At least, that would help explain why he abandoned the Obama administration’s nuclear deal with Iran in his first term—why he wasn’t satisfied with Iran just agreeing not to build a nuclear bomb and submitting to a strict system of monitoring and inspections that guaranteed compliance, but demanded that it go further and do various things that we demand of no other nation. That would also help explain why he chose to attack Iran a month ago rather than persist in negotiations that could have gotten him a deal at least as good as the Obama deal and almost certainly better.

Trump isn’t the only US president who deserves blame here. If President Biden had wanted a restoration of the nuclear deal that Trump jettisoned, he almost certainly could have had that early in his term. But he, like Trump, demanded more. The Iranians, as Biden official Rob Malley explained to me on the NonZero podcast, didn’t buy the idea that they should be the ones to make new concessions even though America was the country that had reneged on the deal in the first place. It wouldn’t have taken a superhuman exercise of cognitive empathy on Biden’s part to anticipate such a reaction. In any event, by the time he reconciled himself to the Iranian view and softened his demands, politics within Iran had shifted toward a harder line. So no deal was done.

Subscribe now

Share

My claim, above, that more robust and widely deployed cognitive empathy could transform US foreign policy, and help us avoid lots of wars, is not the kind of claim that can be convincingly corroborated in the course of a single issue of a newsletter. In fact, it’s a challenging claim to corroborate even in longer form. One reason is that cognitive empathy deficit can impede prospects for peace in so many different ways. Another is that the chains of causality leading to wars can be long and complex, and there’s rarely a single critical juncture where doing a better job of perspective taking would definitely and demonstrably have made the difference.

Still, it’s possible to identify junctures where cognitive empathy could have plausibly made the difference. I’ll close with a quick look at several such junctures in one very long long chain of causality: the chain that culminated in Russia’s invasion of Ukraine in early 2022. At various steps along the way, more cognitive empathy by various actors—Russia, Ukraine, the US—might have made a difference. I’ll confine my examples to ones involving the US.

1. NATO’s vow in 2008, at the behest of George W. Bush, to eventually admit Ukraine as a member. Here the problem wasn’t, strictly speaking, a lack of cognitive empathy. US Ambassador to Moscow Bill Burns had plenty of that; he saw how threatening Moscow would find the prospect of NATO membership for Ukraine. He even predicted that a commitment to such membership could lead Russia to cause trouble in Crimea and the Donbass, which indeed happened. The problem, rather, was that Burns’s repeated attempts to convey this perspective to the White House—he even wrote a memo titled “Nyet means Nyet!”—were unavailing. So Bush strongarmed reluctant European leaders into making the commitment to Ukrainian membership. And this commitment thereafter colored Moscow’s views of developments in Ukraine. For example:

2. The Maidan Revolution of 2014. In retrospect, President Obama might have realized that, if the US conspicuously and vocally supported a protest movement that aimed to depose a Russia-friendly Ukrainian president—and especially if that goal was realized violently, as it was—Vladimir Putin might start feeling insecure about the long-term lease that entitled Russia to maintain a key naval base in Crimea. And, since most Crimeans actually preferred the prospect of Russian rule to the reality of Ukrainian rule, Putin might snatch Crimea almost effortlessly. Which he did, heightening the tensions that eventually led to war.

3. The original decision in the 1990s to expand NATO. As with the 2008 vow to admit Ukraine to NATO, the problem here wasn’t a lack of cognitive empathy per se. Plenty of Russia experts warned that NATO expansion was a bad idea. George Kennan, the most famous of those experts (known, somewhat misleadingly, as the architect of the Cold War doctrine of containment), said: “I think the Russians will gradually react quite adversely and it will affect their policies. I think it is a tragic mistake… This expansion would make the founding fathers of this country turn over in their graves…. Of course there is going to be a bad reaction from Russia, and then [the NATO expanders] will say that we always told you that is how the Russians are—but this is just wrong.”

Bill Clinton was surely aware of this view. But for whatever constellation of reasons—perhaps including such considerations as donations from arms makers and Polish-American votes in the Midwest—he chose not to act on it.

This points to a challenge: Even when national leaders manage to see the national interest clearly, with full illumination from cognitive empathy, they need to resist the temptation to sacrifice that interest to their own political interest. And that kind of virtue, as I suggested last week, doesn’t seem to be in abundant supply.

Share

Share

Banners and graphics by Clark McGillis.

Several decades ago, I learned something that surprised me: Game theorists don’t consider war a zero-sum game.

In retrospect, I’m surprised that I was surprised. I already knew that nuclear war is a non-zero-sum game (since the outcome can obviously be lose-lose—as in, everybody dies). But I guess I’d thought of nuclear war as some kind of unique and aberrant subspecies of war, rather than what it is: just a particularly pronounced example of the fact that wars often lead to massive death and destruction on both sides. Does it really make sense to say that the “victorious” nations in World War I, having lost at least 10 million lives, were better off than before the war started? Is it even clear that they fared better than the “losing” nations, which lost a bit more than half that many lives? By what calculation can you say that the sum of the fortunes of the two sides was zero rather than subzero?

I didn’t initially welcome this news flash from game theory. At the time, I was starting to work on a book (which wound up being called Nonzero), and, in ways that aren’t worth getting into, the non-zero-sumness of war complicated an appealingly simple narrative framework I’d been planning to employ. But, that inconvenience aside, the fact that war is non-zero-sum seems like potentially good news. If nations rationally pursue their self-interest, shouldn’t the knowledge that war often makes both sides worse off discourage them from starting wars?

In theory, maybe. But, back in the real world, there’s a massively destructive war going on in the Middle East.

Well, we might as well put it to good use! I think viewing the Iran War in game theoretical terms can shed light on the question of why humankind seems so bad at respecting the logic of game theory—why nations keep getting into wars that, history tells us, may inflict huge costs on all concerned.

Share

Subscribe now

But to get that kind of illumination, we need to first add a little complexity to our game theoretical model—two dimensions of complexity, to be exact.

The first dimension is one that may have already occurred to you—as a kind of objection to how simple my initial explanation of war’s non-zero-sumness was. I was acting as if the consequences of war are confined to the number of lives lost. And there’s more to life than lives! There’s territory, for example. Right now Israel is seizing Lebanese territory after driving off Lebanese residents—and if the past is any guide, Israel may hang on to the territory. And every acre it gains is an acre Lebanon loses: zero-sum.

Another kind of benefit of war is lasting security gains. Benjamin Netanyahu would probably argue that holding on to that Lebanese land enhances Israel’s security by creating a buffer zone, and he’s definitely argued that inflicting havoc on Iran will enhance Israel’s security. Whether or not these things turn out to be true, it’s true that they could be true; there is such a thing as enhanced security, and sometimes it’s achieved by war.

So if you add this dimension—if you expand your tabulation of war’s consequences beyond loss of life, to include things like land and future security—it’s possible for a nation to come out ahead even after accounting for fatalities and other human casualties. Which means it’s possible, in principle, for a war to have a win-lose outcome, even if wars are often lose-lose. So maybe it’s not so irrational for nations to keep embarking on war after all.

Maybe. But, if you want to explain why war is such a stubbornly persistent feature of human life, I don’t think this particular dimension of added game theoretical complexity is the ticket. I’d put more emphasis on the second added dimension. And, actually, maybe “dimension” is a misleading term here. Maybe you should think of this not as a dimension but rather as a whole second level of accounting—the level of domestic politics. To put it another way: The interests of the people who decide whether to launch wars are different from the interests of the nations they lead. A nation’s leaders can benefit politically from a war even if the nation, on balance, doesn’t benefit from it.

Consider Bibi Netanyahu. There’s no doubt that he’s benefited politically from Israel’s being so often in a state of conflict. Before October 7, 2023, for example, he was deeply unpopular and was spending part of his political energy trying to wriggle out of criminal prosecution. But, even though October 7 represented an epic security lapse that was clearly his fault, he masterfully turned it into a get-out-of-jail free card. By sustaining the war on Gaza well past the point where Hamas had been effectively neutralized, he rose in the public’s esteem. And then last summer, in the final months of the war, he started a whole new conflict—with Iran—that was deemed a triumph and helped keep him popular even after his rate of killing Gazans had dropped from wartime levels to mere ceasefire levels.

Will the scorched earth policy he pursued in Gaza—and is now pursuing in Iran and Lebanon—be good for Israel’s security in the long run? I’m skeptical; I think that if Israel wants lasting security, increasing the number of people in the Middle East and the world more broadly who hate it, and the intensity with which they hate it, isn’t the optimal strategy. But I could be wrong about that. In the annals of history, the ruthless deployment of brute force has proved effective with disconcerting frequency.

In any event, I think it’s clearer that Bibi’s belligerent policies have been good for Bibi than that they’ll be good for Israel in the long run. And however far back you go into his political history, you see the same pattern: He’s a deft sustainer of tension and conflict with Israel’s enemies. He has done his best to sabotage a two-state solution, and he has at times maneuvered to boost Hamas at the expense of more moderate elements in Gaza. And, conveniently, these two goals have sometimes dovetailed. According to the Israeli newspaper Haaretz, Netanyahu, in a 2019 meeting with Likud members of the Knesset, explained why he tolerated the Qatari government’s financial support of Hamas: “Anyone who wants to thwart the establishment of a Palestinian state has to support bolstering Hamas and transferring money to Hamas. This is part of our strategy—to isolate the Palestinians in Gaza from the Palestinians in the West Bank.” One effect of all of this was to sustain and heighten the sense that Israelis were under siege—much to Bibi’s political benefit.

Share

Same goes for Iran’s political leadership: It has benefited politically from having enemies. The late Supreme Leader Ali Khamenei found it politically useful to depict America and Israel as implacable foes.

Of course, over the past few weeks, America and Israel have been acting like implacable foes of Iran—so Khamenei doesn’t seem to have been fabricating enemies out of whole cloth. And you could say something similar about Netanyahu: Yes, he’s played up the threats from Hamas and Hezbollah, but the threats themselves were real.

This fact—that the external threats that national leaders can profitably highlight and hype tend to have some basis in reality—makes it hard to isolate the forces that drive nations into war. Often when a nation initiates a war you can come up with both a plausible national security rationale and a plausible domestic political rationale. And it’s hard to say which played a bigger role.

One great thing about President Trump is that he takes the guesswork out of this exercise. There was no remotely plausible national security threat that justified attacking Venezuela, and the same goes for his attack on Iran. His claim that there was some kind of nuclear threat is ridiculous. Iran was nowhere near having a nuclear bomb and had just expressed a willingness to remain verifiably nuke free—and to make more concessions as part of a new nuclear deal than it made as part of the perfectly adequate Obama nuclear deal, the one that Trump catastrophically trashed during his first term. In fact, it was during the negotiations that had revealed this willingness on Iran’s part that Trump launched his surprise attack on Iran three weeks ago.

In the absence of a plausible strategic rationale, we’re forced to conclude that Trump launched this war because he thought it would be a political winner. Sure, he probably told himself it would also work out fine for America. But for a president to think that such a war actively and importantly served America’s national security interests, he’d have to be more clueless than I think Trump is, which would be quite a feat.

Subscribe now

Why exactly Trump thought attacking Iran would be a political winner—how he came to think this was going to be Venezuela all over again—is unclear. Was he convinced of this by Bibi or Mark Levin or Marco Rubio or Lindsey Graham or…? Hard to say what the constellation of key influencers was, but in any event their mission wasn’t exactly mission impossible. Trump isn’t a complicated man, and he’s never had a deep understanding of the world, and he seems increasingly detached from reality.

Anyway, how exactly this fiasco took shape isn’t our subject today. Our subject is the more generic one of why wars keep happening. And my point is that the answer lies at least as much at the level of domestic politics as at the level of geopolitics and national interest. At the level of domestic politics, wars can easily make sense; they can be win-lose or even win-win. For more than 15 years, Netanyahu and Hezbollah leader Hassan Nasrallah benefited politically from each other’s enmity—until finally Netanyahu’s enmity proved lethal, and the relationship shifted from win-win to win-lose.

I think highlighting the fundamentally political, as opposed to geopolitical, causes of wars could make them less common. The more clearly people understand where the real motivation usually lies, the more inclined they’ll be to greet the next push for war with appropriate cynicism.

I listen to a lot of podcasts, and some of them are what you could call foreign policy establishment podcasts—they’re produced by, say, the Council on Foreign Relations or some very buttoned up DC think tank, or they feature conversations among the kinds of people who work at such places. And, almost invariably, the people on these podcasts, in gravely assessing the motivations that start and then steer wars, stay at the level of geopolitics and national interest and assiduously avoid the level of domestic politics. To hear them talk you’d think that Trump was Metternich—or at least a dimmer version of Metternich—rather than a former Reality TV star who is just trying to keep his ratings up by staging a new spectacle that’s more eye-catching than the last one.

This kind of credulous discourse is a disservice to the nation. It sustains the myth that the people who steer American foreign policy are by and large worth taking seriously. They’re not. The politicians who steer it are for the most part just trying to get re-elected—and will serve whichever cluster of special interests can further that cause. And the “experts” who help steer it, including many of the voices on these podcasts, are people who managed to get hired by think tanks that, for the most part, are funded by the same special interests that are corrupting those politicians.

The better people understand all this—the more cynical they are about their political leaders and opinion leaders—the less susceptible they’ll be to the endless threat inflation that is the lifeblood of the military industrial complex and the various narrower lobbies that work hand in glove with it.

So, yes, war among nations is a non-zero-sum game, a lose-lose endeavor more often than not. But for the national leaders who keep pushing us into those wars, there’s more upside. We need to change that.

Share

This week on the NonZero podcast Daniel Levy, who has worked as a negotiator under two Israeli prime ministers, talked about, among other things, the increasingly religious character of Israeli politics and how that has influenced foreign policy:

Share

Banners and graphics by Clark McGillis.