On Thursday of this week President Trump was scheduled to sign an executive order outlining a process by which the government would review the most powerful new AI models before their release. This was less momentous than it sounds. Participation by the companies that make these models would be voluntary, and the executive order explicitly ruled out any actual government regulatory power: “Nothing in this section shall be construed to authorize the creation of a mandatory governmental licensing, preclearance, or permitting requirement for the development, publication, release, or distribution of new AI models.”
But apparently even this degree of oversight was too big a burden for Silicon Valley to bear stoically. David Sacks, who recently stepped down as White House AI Czar but now gets to play Rasputin, phoned Trump Thursday morning and lobbied against the order. By some accounts, Elon Musk and Mark Zuckerberg chimed in as well. Hours before the signing ceremony, the White House announced its indefinite postponement. A new variant of TACO—a variant in which Trump is on the verge of defying billionaire AI accelerationists before he chickens out—had been born.
Explaining his decision, Trump flawlessly recited the central anti-regulation talking point of the accelerationists: “We’re leading China, we’re leading everybody, and I don’t want to do anything that’s going to get in the way of that lead.”
The reaction to Trump’s pivot fell along predictable lines. AI safety advocates lamented that even the most tentative step toward government monitoring of the AI industry had proved, for the time being at least, untenable. Voices of the AI industry replied that this tentative step could have been the camel’s nose under the tent. The Washington Post—owned by Amazon founder Jeff Bezos—lauded Trump’s decision as “prudent.” Granted, said the Post’s editorial board, the executive order “was limited to cybersecurity”—the government had apparently just wanted to use its sneak preview of powerful models “to ensure that any critical vulnerabilities adversaries could exploit in the nation’s critical infrastructure were promptly patched.” But “that scope could easily have broadened beyond cybersecurity to include more nebulous ‘safety’ concerns favored by advocacy groups.” All told, “The president is right to be wary of creating a bureaucratic review regime that erodes America’s edge.”
If forced to choose sides here, and align either with Silicon Valley accelerationists or the safety crowd, I wouldn’t hesitate. I don’t see how anyone who senses the emerging power of this technology—something you’d expect Silicon Valley billionaires to be capable of—could believe that faster is better, period. It seems to me that, in rejecting any and all government oversight, the accelerationists are being either deeply cynical or bafflingly naive. Either way, they must be stopped. If they keep sabotaging attempts to exert meaningful influence on our technological future, we’ll be in deep trouble. People in the AI safety community may have some mistaken ideas of their own—they disagree with one another enough that some of them must have mistaken ideas—but at least they evince a sense for the magnitude of what’s unfolding, an appreciation of the stakes.
Still, to end the conversation there, with an endorsement of the tiny step toward oversight that Trump almost took, would be to give the issues raised by this week’s episode of TACO short shrift. This isn’t just another left-right debate about the virtues of deft regulation and the perils of ham-handed overregulation; AI has unique properties that make the issue much more complicated than that. To put it another way: If I just endorsed this modest executive order without elaboration or qualification, I’d be doing what I’m accusing the accelerationists of doing: failing to exhibit due appreciation of the stakes. The fact is that there really is something scary about giving Trump even modest oversight of AI—it’s just not the thing that Sacks and other prominent critics of the executive order are worried about.
Consider this thought experiment:
Let’s assume that, contrary to the Washington Post’s fears, the practical scope of this executive order remains confined to the realm of cybersecurity. In this scenario, the Trump administration could wind up with access to an unreleased model that has cyber capabilities far beyond those possessed by any AIs already on the market.
And actually, come to think of it, this isn’t just a thought experiment. The administration does now have access to the still-unreleased Mythos, Anthropic’s legendarily powerful tool for detecting—and in principle exploiting—cyber-vulnerabilities. Which raises a question:
What is the government doing with Mythos? Well, given what we know about this administration, there are unsettling possibilities that can’t be entirely ruled out. Is Pete Hegseth preparing to use Mythos to take down Cuba’s power grid, thus adding to the crimes against the Cuban people that the administration is already committing with its blockade of the island? Or maybe the Trump administration will use Mythos to penetrate China’s infrastructure in a way so unprecedentedly threatening that, if the Chinese government finds out about it, rash action might ensue? I very much doubt either of these things will happen—but given the wantonness with which this administration violated international law in attacking Venezuela and Iran, I don’t feel wholly confident that it’s not doing something lawless and/or reckless with Mythos.
And what about the domestic front? Imagine that, a year or two from now, the administration has access to a model so far ahead of the curve that, in the runup to an election, it could flood social media with undetectable and highly persuasive bots? Can we be sure that Donald Trump, the most blatantly unscrupulous person ever to occupy the oval office, would resist a temptation like that?
Again, these are pretty fanciful examples. My point is just that the regulation of AI is different from the regulation of, say, auto emissions or workplace safety or pretty much anything else. In those cases the government could, through regulatory overreach, stifle initiative or otherwise impede economic efficiency—but it couldn’t start World War III or a covert mass persuasion campaign that subverted the American republic.
Here’s another distinction between AI and lots of other things that get regulated: If you refrain from regulation altogether, that could be not just bad (as it would be with, say, workplace safety) but catastrophic on a planetary scale. The most famous example, but not the only one, is that unregulated AIs could help someone create a bioweapon that started a pandemic.
In sum: Both ends of the regulation spectrum—powerful government control at one end and pure laissez faire at the other—pose much bigger dangers with AI than with most things that get regulated. So AI gives us a lot to think through!
That fact wouldn’t be so unsettling if we had lots of time to do the thinking. But one thing this week, like so many recent weeks, drives home is that we don’t. Consider, for example, some of the week’s news about Anthropic:
As of last summer, Anthropic was telling investors it didn’t expect to reach profitability before 2028. This week the company released revenue numbers that change that picture. As a Wall Street Journal headline put it, “Mind-Blowing Growth Is About to Propel Anthropic Into Its First Profitable Quarter.” Since the beginning of last year, Anthropic’s annual recurring revenue (monthly revenue times 12), has gone from around a billion dollars to around $40 billion. People who know about these things say revenue growth at this rate, and on this scale, is unprecedented—like, in the entire history of capitalism.
And note that, by virtue of Anthropic’s business model, the revenue comes mainly from companies, not consumers—which means that Anthropic AIs are doing a lot of work that human workers have traditionally gotten paid to do. So continued Anthropic revenue growth could mean real impact on the labor force, for better or worse.
And then there’s this: One company where work is being thus automated is Anthropic. So the kinds of AI tools that Anthropic is building accelerate the building of new kinds of AI tools. This kind of positive feedback cycle is what, in theory, could lead to the “singularity”—the point at which change starts happening so fast that God only knows what lies beyond it.
This week at an annual Google conference known as I/O, after the company had outlined plans to sprinkle AI onto pretty much every product it makes, Demis Hassabis, head of Google’s Deep Mind, turned heads by musing, “When we look back at this time, I think we will realize that we were standing in the foothills of the singularity.” On that same day, Anthropic said that Andrej Karpathy, a co-founder of OpenAI and the former head of AI at Tesla, would be joining Anthropic, leading a team that uses Anthropic’s large language model, Claude, to accelerate research on the training of LLMs. In other words: Karpathy’s mission is to accelerate Claude’s self-acceleration.
Well, at least for now there’s a human in that loop. Maybe we should make some progress on this AI regulation issue while that’s still the case.
Note: Publication of my book on AI, The God Test, is a month away. Maybe you should pre-order a copy now in case the singularity disrupts the world’s electronic payment infrastructure before June 23!
As noted in last week’s Earthling, my view is that the whole world benefited from the Trump-Xi summit. With that in mind, you might think I consider the framing of the survey below—“who benefited more”—unfortunately zero-sum. Well, strictly speaking, no; even in a non-zero-sum game that has a win-win outcome, it’s possible for one side to benefit more than the other. Still, the wording of this question does probably steer some people toward a zero-sum way of thinking about the summit’s outcome. Which I guess means people who replied that Trump and Xi benefited by an “equal amount” deserve credit for nonetheless giving an unambiguously non-zero-sum answer. And—credit where due—MAGA supporters were more likely to do that than any other group.
Banners and graphics by Clark McGillis.