Below is Part II of a five-part essay that I’m rolling out on NZN. (Part I appeared last Tuesday.) The essay is a kind of call to action—an argument about what we need to do to successfully navigate the AI revolution and why we need to do it. If you’d like to help me get this message out, please share (and/or like) my tweet about it or my bluesky post about it. Thanks!
Part II: The roots and branches of acceleration
It’s long been true that technological advances could facilitate subsequent technological advances. This dynamic is especially common in the realm of information technology, where advances often make collaboration among scientists and engineers more far flung and more efficient. The internet was only the latest in a long line of such innovation accelerators.
In the 1960s, the mathematician I.J. Good imagined an era when the self-reinforcing advance of information technology would make a quantum leap—the era of artificial intelligence. A sufficiently intelligent machine, he wrote, “could design even better machines.” And the resulting machines could repeat the process—so “there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind.”
It was the science fiction writer Vernor Vinge who fused Good’s musings with John von Neumann’s use of the term singularity. In a 1993 paper that cited both men, he wrote, “When greater-than-human intelligence drives progress, that progress will be much more rapid. In fact, there seems no reason why progress itself would not involve the creation of still more intelligent entities—on a still-shorter time scale.” He suggested that, “it’s fair to call this event a singularity (‘the Singularity’ for the purposes of this paper).” The Singularity, he wrote, would be “a point where our models must be discarded and a new reality rules.”
This sounded like pure science fiction at the time—and, actually, for decades thereafter. But in early June of this year, the AI company Anthropic published a paper that, though it didn’t mention the singularity, brought to mind the title of Kurzweil’s book: The Singularity is Near.
For the past few years, Anthropic said, each generation of its large language model, Claude, has been playing a role in developing the next generation’s model—and, moreover, a larger role than the previous generation had played. This isn’t just a matter of LLMs writing more and more of the computer code. Though the shifting of this burden from human to machine has been dramatic, more notable is the growing ability of LLMs to participate in the research process—doing experiments to see what innovations could make the next generation better than the last.
Claude, says Anthropic, “can already match or outperform skilled humans at executing a well-specified experiment.” The more challenging part of the research chain is “deciding what experiments to run, interpreting what comes back, and figuring out which ideas to try next.” And, though in this area “large performance gaps” between human and machine persist, they’re shrinking. Anthropic says that in April, “Claude-powered agents were given an open problem in AI safety—roughly, can a weaker model reliably supervise a stronger one?—and were left to solve it. This involved proposing hypotheses, testing them, sharing findings with parallel agents, and iterating.” The agents got some guidance along the way but they “designed every experiment themselves.” An engineer who provided that guidance said the agent’s performance was on par with that of a good junior colleague and declared, “The future is now.”
Well, it depends on what you mean by the future. In the strictest singularity scenario, this engineer would no longer have a job. We would have reached “recursive self-improvement,” the point where the machines just keep building better versions of themselves, with no humans in the loop. But according to the Anthropic paper—titled “When AI builds itself”—even that point may not be far off. Current trend lines point to “an AI system capable of fully autonomously designing and developing its own successor.” The paper continued, “We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for.” And one consequence might be to “increase the risks of humans losing control over AI systems.”
This may sound alarmist—and there are people who accuse Anthropic of self-servingly hyping AI risk—but in a sense the company, by focusing narrowly on recursive self-improvement, is underplaying the threat we face. The self-accelerating character of AI progress could bring radically disruptive change well before this threshold arrives—and even if it never arrives. We could still reach a point where, as von Neumann put it, human affairs would not continue “as we know them.” In that sense, the singularity could be nearer than Anthropic is suggesting.
In fact, there are signs that it is—signs of more and more dramatic AI advance. Machines have started solving problems that the world’s best human mathematicians had struggled with for decades. Superhacking AI agents like Anthropic’s Mythos have been finding software vulnerabilities that had escaped detection by legions of human engineers. But the most vivid and possibly the strongest piece of evidence that AI is driving us faster and faster toward an epic threshold is something that’s taken shape not in recent months but in recent years: a graphical depiction of AI progress developed by a nonprofit called METR—a depiction that, within the AI community, has reached iconic status.
METR’s researchers measure how long it would take a human to do tasks that the most powerful large language models can do, and in early 2025 they reported a pattern that had persisted since the dawn of the LLM age six years earlier: The human “task time” that these models can match had been doubling roughly every seven months. That’s an exponential growth rate that, when plotted on a graph, looks like it’s “going vertical” (a common feature of singularity-signifying curves from math and physics). And as if that weren’t enough: Since 2025 the doubling time has been shrinking; the curve is going really vertical.
This curve represents improvement in various AI skills—and, most important, in a kind of meta-skill: autonomy. Autonomy is the ability to pursue an assigned goal flexibly, recognizing and overcoming obstacles, backtracking and retooling in the face of failure. Whether you’re trying to prove a math theorem, research a history dissertation, or do various kinds of jobs that AI agents may encounter in the workplace, autonomy is critical to success. That’s why the market is encouraging the big AI companies to build more and more autonomous AI agents—guidance the companies are following with furious intensity.
Unfortunately, AI autonomy can be dangerous. It’s the reason that in July hundreds of AI agents were able to escape the supposedly secure “sandbox” that OpenAI had put them in for evaluation and break into computers at a company called Hugging Face, looking for information that could help them get a good grade from the evaluators. They pursued their assigned goal more flexibly than the goal-givers had anticipated.
The reaction to the Hugging Face incident was voluble and, if you share my vision of the future, heartening. Not only did the burst of publicity spread awareness of the dangers of rampant AI autonomy; the reaction within the AI community signaled awareness that the response to such dangers will naturally carry governance beyond the national level, to the international level.
Coming Friday, Oct. 2: Part III: The inexorable logic of international governance
Meanwhile: If you want to check out my book on AI, The God Test, you can read the introductory chapter and excerpts from all other chapters at thegodtest.net.]
Banners and graphics by Clark McGillis.
Man, I would love to host you and Cory Doctorow in a discussion on this. Is he really misfiring this time in poohpoohing almost all of this as hype/marketing?