

Hi! Yesterday I published a piece in the Washington Post arguing for a government-mandated AI slowdown in the US. If you like lean, tightly-edited pieces, you should read it—but if you’d prefer a longer, meandering version with 45 percent more fat, then you’ve come to the right place! That’s what you’ll find below.
Also: I had a great Zoom chat Saturday with NonZero members about AI. That conversation is now available on the NonZero podcast feed (the members’—aka paid subscribers’—edition of the feed). I encourage you to listen to it, but even if you don’t you can still ponder a question I posed during the conversation: How can we use the NonZero platform to constructively influence the discourse about, and/or policy about, and/or use of, AI—maybe even playing some small role in helping to keep the world from going off the rails? And, to that end, should we try to get NonZero readers and listeners engaged in the process—maybe at first via discussion groups and then via whatever grows out of them? I’m just thinking out loud here, but if you find the thoughts intriguing you can email us at nonzero.news@gmail.com—and/or share your thoughts in the comment section below.

As Anthropic approached this week’s public stock offering, the company persisted in its famously paradoxical messaging—warning about the dangers of rapidly advancing artificial intelligence while telling us how rapidly its artificial intelligence is advancing.
A week ago, in an amendment to its S-1 filing with the SEC, Anthropic said it had now reached a threshold described in its safety policy as a “dramatic acceleration” of AI progress attributable to the growing role of AI itself in creating new models. In the original filing, it had warned that such acceleration could “cause the capabilities of our models to advance more rapidly than our ability to develop, validate, and implement corresponding safety, alignment, and security measures.” Also: Anthropic may not be able to identify “a significant acceleration in our rate of development until after it has occurred.”
Anthropic made this revision to its S-1 a week after METR, a highly regarded NGO that assesses the capabilities of AI models, issued a report about Anthropic that noticeably raised the freakout level within the AI safety community. The report found that Anthropic’s best models have over the past year advanced roughly twice as fast as during the preceding year and a half. This is the kind of steepening slope of progress that is said to precede “recursive self-improvement”—the point where the AI is designing and building new generations of itself, with humans relegated to a marginal role if any.
Paul Christiano, a former head of alignment at OpenAI who has made fundamental contributions to AI research, said on a podcast that the METR data was “exactly the data you would expect to see 6-12 months before producing superintelligence”—which means, he said, that there is a “reasonable chance” that “in six months, there’s a disaster that kills very large numbers of people, or where the government is overthrown.”
Those last five words may sound crazy, but this kind of talk is not uncommon in AI circles. In fact, in 2023 Anthropic CEO Dario Amodei, appearing on the same podcast, said that someday we’d have powerful agentic models and “if such a model wanted to wreak havoc and destroy humanity or whatever, I think we have basically no ability to stop it.” We “will reach the point where [this is] true, as we scale the models... That definitely seems the case.”
Well, now that we do have powerful agentic models, why does Amodei keep scaling them? Why doesn’t Anthropic slow down?
One answer can be found in a letter posted by Amodei two months ago, a letter that emphasized the value of slowing down—and that some people took as a promise that his company would slow down. Maybe these people didn’t read the fine print. More than a dozen paragraphs after declaring that “we must slow the pace at which we improve the capabilities of AI models,” Amodei added that American AI labs shouldn’t slow down enough to endanger “the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party.”
Amodei’s concern with winning the AI race against China is shared by many in Silicon Valley and Washington DC. That’s one reason for the near-consensus that any significant slowdown, such as a pause in the large training runs that create new generations of AIs, must be “internationally coordinated.”
But for political and geopolitical reasons, this prospect seems remote—all the more so now that the scandal which put Nvidia CEO Jensen Huang out of a job has also put wind in the sails of US China hawks and raised the political cost of cooperative overtures toward Beijing by President Trump.
The time may come when an effective slowdown indeed requires international coordination—and certainly the AI challenge will eventually demand various international policies. But right now, a national slowdown—such as a government-mandated pause of large training runs—is in the US interest. It would not pose the threat to American AI supremacy, or to American economic vibrance, that many claim, and it could pave the way for the kind of international cooperation on AI that now seems distant.
Though China’s best AI models are said to be six to nine months behind American models—and a straightforward comparison of capabilities suggests as much—there are two reasons that’s misleading.
First, as Amodei has tirelessly pointed out, leading Chinese AI companies rely partly on the “distillation” of the strongest American models. They give these models lots of prompts, record the responses, and then use those input-output pairings as training data. Distillation saves time and computing power and, maybe most important, can capture the practical benefits of research advances made in American labs.
So, to the extent that Chinese AI progress depends on distilling the best American models, that progress would slow down in response to an American slowdown. The metaphor sometimes used to capture this dynamic—American companies are motorboats and Chinese companies are water skiers—overstates the dependence for the sake of vividness, but the core of the metaphor is apt.
Meanwhile, even if the gap between the most lucrative American models and rival models narrowed somewhat, a hidden asset of frontier American AI companies would help ensure their commercial success. Anthropic and OpenAI are known to have unreleased models that are much more powerful than their released models—and that, since they aren’t available for distillation, are well ahead of Chinese unreleased models.
These models will supercharge a little-noted part of both companies’ business strategies—developing new drugs and other patentable inventions and then profiting from the intellectual property. Anthropic has bought two biotech companies this year. And both companies are collaborating with big pharmaceutical firms, as is Google. The stunning math breakthroughs recently made via unreleased models—in great profusion at OpenAI and on a smaller scale at Anthropic—hint at the competitive advantage such models could confer in other domains.
What’s more, frontier AI labs could, without actually releasing their most powerful models, and thus exposing them to distillation, still charge big American companies very big sums to make use of them. OpenAI is known to have approached American companies with this proposition.
The commercial potential of unreleased models helps explain why some people suspect that OpenAI and Anthropic would actually welcome a pause if it were mandated by the government and so applied to all their American rivals. These suspicions may or may not be right, but their plausibility supports the expectation that a pause in large training runs would permit continued commercial dynamism at the American AI frontier.
And, as Wharton professor Ethan Mollick and others have noted, if the creation of new generations of models stopped now, the productivity gains from AI in the broader economy would accumulate for years as American companies figured out how to adapt the technology to their needs, aided by new apps and “agentic harnesses” crafted for existing models.
The advantage enjoyed by America’s leading AI companies over Chinese rivals will be strengthened by some likely consequences of the Nvidia scandal: stricter enforcement of rules banning the export of high-end chips to China and new rules that limit the ability of Chinese companies to rent processing power from data centers outside of China. In addition to handicapping Chinese AI companies, this tightening could, by shrinking the market for advanced chips, put downward pressure on the prices US companies pay for them.
President Trump is currently opposed to a government-mandated pause—and for that matter to pretty much any form of firm AI regulation. But skepticism about AI on the right—including in large swaths of MAGA—may mean that some forms of AI regulation, including a mandated pause, could pass the next Congress with a veto-proof majority. And, in any event, Democrats should pass the best legislation they can and then see if President Trump, amid great public anxiety about AI, is willing to veto it.
In the long run, the most important byproduct of a national pause could be an international pause. Many Chinese elites, in and out of government, think that calls in Washington and Silicon Valley for an international slowdown are just an attempt to solidify America’s AI lead—a crafty complement to the chip export controls that explicitly serve that goal. A unilateral US pause could help dispel these suspicions and, as the risks of AI become clearer to both governments, pave the way for an international slowdown.
If there’s no slowdown of any kind in the coming months—if US frontier labs continue to advance at their current pace, while newly tightened chip export restrictions hem China in more fully—the result could be explosive. The bigger the gap between Chinese and American AI, the more threatened Beijing will feel. If America’s AI advantage seems to be approaching a point of clear strategic dominance, pre-emptive Chinese action could well ensue.
This could mean cyberattacks on US data centers or AI labs, or a military attack aimed at seizing or destroying the Taiwanese factories that make the chips China isn’t allowed to import, or various other assaults that could lead to a war between nuclear powers.
The race toward artificial superintelligence may be one of those rare races in which slowing down is the only way to come out a winner.
You can read the introductory chapter of my recently published book on AI, and excerpts from all other chapters, at thegodtest.net.
Banners and graphics by Clark McGillis.

Keeping unreleased models private protects lab profits during a pause. Still, wouldn't Beijing read that hidden stash as a head-fake instead of goodwill? In The Evolution of Cooperation, Robert Axelrod ties lasting restraint to transparent, verifiable moves (https://www.amazon.com/dp/0465005640).
Hi,
I know there is at least one fairly large group in the US … called Third Act … organized in 2021 by Bill McKibben. The Third Act group was built around climate change, democracy, and racial justice.
I wonder what Mr. McKibben has learned, and would be willing to share about building a group designed to support/promote a cause, and take some action? What other groups does anyone else know about that might have organizational ideas that would suit our needs? Maybe we don’t necessarily have to reinvent the wheel!
Also I wonder what “ Non-zero platform “ means? Bob mentioned that during the zoom. Do we mean Mission/Vision Statement, and goals…… or where people gather … like discord or … some other organizational structure?
Thanks,
Noreene