Blog

The mismatch is striking. On one side, warnings of existential risk, maybe a 10% chance of the end of civilization. On the other, government paralysis.

The drumbeat about unpredictable, out-of-control artificial intelligence agents increased dramatically in the past several weeks. Washington not only struggled to respond. It barely could agree on whether there was much of anything to worry about.

It started with AI agents that not only escaped their confines, but appeared to conspire among themselves to hide their tracks as they hacked into another firm’s library of AI tools.

Then, in a deliberate echo of Albert Einstein’s letter to Franklin D. Roosevelt about the potential power of nuclear weapons, more than 1,300 computer scientists called for the industry to slow down, and for the government to step in.

In recent days, Jacob Coxon, an Anthropic engineer who previously worked for OpenAI, resigned in the most public way, declaring on social media that the two leading AI labs were professing deep concern while still speeding toward “superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.”

On Saturday, Dario Amodei, the CEO of Anthropic, released a 3,800-word essay calling for a global slowdown of AI development, echoing more urgently concerns he had raised in the past. Elon Musk, the billionaire rival in the AI race who remains close to President Donald Trump, quickly responded: “Dario is right.” Sam Altman, the CEO of OpenAI, also chimed in with support.

Yet in Washington, there has been almost no substantive response. If the 1940s through the 1980s were consumed by finding ways to reduce the risk of a nuclear exchange — creating new institutions like the Atomic Energy Commission and arms control treaties to govern how the world’s most dangerous technology would be handled — the approach of the U.S. government so far has been the lightest possible.

There are many computer scientists, of course, who believe these warnings are overinflated. Trump himself has been at the forefront of the “what-me-worry?” crowd, refusing to engage on how to balance the risks with the possible rewards. On Thursday, asked by a reporter about whether he had “any concerns about AI leading to human extinction,” he brushed it off.

“No, I don’t have any,” he said. “I have concerns that if we don’t win AI, we’re going to be put in a very bad position.”

The president quickly turned to the answer the White House — and those who view the calls for slowdowns and regulation as a huge strategic error — sees as the deciding factor: If the United States falters, China will take its place.

“We are leading China right now by a pretty good period,” Trump said, noting that Xi Jinping, China’s leader, was coming to Washington in less than two weeks. But rather than talk about exploring a bilateral AI accord with the Chinese, he turned to the state dinner that he said would be a highlight of the visit. “I think it’s going to be great,” he added.

“Great” is not the word most used in this summer of AI surprises.

The apocalyptic scenarios that are most feared in Silicon Valley include AI systems becoming so advanced that they learn to automatically improve themselves, a process called “recursive self-improvement.” A related concern, arising from a series of incidents at OpenAI and other AI labs, is that agents break free from constraints and decide to complete their programmed tasks in a way that is “misaligned,” in AI industry speak, with human intentions. Another worry is that AI could enable rogue bad actors — or reckless governments — to create deadly pathogens, gain access to nuclear command and control systems, or unleash unstoppable cyberattacks that threaten major loss of life or societal collapse.

Computer scientists suggest that Americans stop thinking about AI as a mere tool, but akin to an alternative species, one the world will have to live with, for better or worse.

The White House declined to make its key AI policymakers available for questions. But it said that the administration continued to balance innovation and security, deflecting any specifics.

(The New York Times has sued OpenAI and Microsoft, claiming copyright infringement of news content related to AI systems. The two companies have denied those claims.)

Trump signed an executive order early in the summer creating a voluntary system for frontier AI labs to submit their new models for a 30-day safety review. But it was full of loopholes. (Open source models, the kind that Chinese firms are pumping out and that can be tailored to specific needs by companies, scientists, or, perhaps, terror groups, are not subject to the review.)

Amid the many memorial events on Friday for the 25th anniversary of the 9/11 attacks, the contrast with that era is notable. The revelation of U.S. vulnerability led Vice President Dick Cheney to develop what soon became known as “the 1% doctrine.” Prompted by fears that Pakistani scientists could help al-Qaeda develop a nuclear weapon, it posited that government had to act even if the risk was remote, because the consequences would be so high.

In this case, 1% looks like pretty good odds: Evan Hubinger, who leads the science of “alignment” at Anthropic, which tries to contain the code to align with human goals, wrote in recent days that the risk of mass extinction over the next decade was above 10%.

It is impossible to know whether that estimate is right. But as Ben Buchanan, a Johns Hopkins assistant professor and the co-author of a forthcoming book, The Bitter Struggle: Superintelligence, Superpowers, and the Fate of the World, put it: “There’s no doubt in my mind that the technology is going to continue to get much better very quickly, raising urgent questions about biological risks and loss of control.”

“We are in a period of exponential development, and there are only two ways to respond: too early or too late,” he said. “The world has missed its chance to be too early.”

Fear of getting ahead of Trump

Trump’s blase attitude about existential AI fears influences both his administration and the Republican Party he still tightly controls.

At a conference in Washington last week, government officials and corporate executives talked about the promises of AI — for everything from next-generation military operations to espionage to cybersecurity in critical infrastructure — during nearly every panel. While the risks of AI wielded by adversaries were mentioned at times, what was little discussed was the need to slow down or regulate the powerful technology.

Speaking on stage last Thursday at the Billington Cybersecurity Summit, Sean Cairncross, the White House national cyber director, said the administration was “trying to balance the innovation side and running at speed with securing our systems and handling this technology responsibly, which is to say, not letting it fall into the hands of people who would do us harm, our adversaries.” Cairncross did not directly address ascendant fears about the rapid pace of AI advancement, or recent episodes of rogue agents breaking loose and hacking into other networks.

Some in the White House and elsewhere in the administration do see a need to be more proactive in addressing AI safety risks, but do not want to cross Trump, former officials said.

“There is a fear of getting ahead of the president,” said Michael Garcia, who served as the associate chief of policy at the Cybersecurity and Infrastructure Security Agency until he departed in June.

“We’ve seen tech executives call the president when they hear about a potential new policy to voice their displeasure, and that then leads to a policy changing or being killed,” Garcia said. “This inevitably leads to inertia to develop new national security policy that may be sorely needed.”

Indeed, David Sacks, a venture capitalist and Trump’s influential former AI czar, dismissed during an interview with Fox News the warnings by Coxon as a coordinated media operation that was “designed to scare the public” and lead to onerous government regulation. And Jensen Huang, the CEO of the chips juggernaut Nvidia, said the new round of fear was ginned up by the cybersecurity industry.

On Capitol Hill, the story is not much different. Until last week, AI only registered for many lawmakers as it related to public antipathy about the construction of data centers, which has become politically salient in the midterm elections, according to lawmakers and aides in both parties. Past efforts to examine AI’s potential consequences have led to hearings and alarmed statements of concern from both parties, but little else.

Rep. Josh Gottheimer (D., N.J.), who sits on multiple AI and cybersecurity subcommittees, said the breakneck speed of AI’s evolution made it difficult for lawmakers to keep up.

“I keep stressing the importance of regularly educating lawmakers on AI and the latest developments,” Gottheimer said. “Recent incidents like the Hugging Face hack are a real warning sign for what can happen if we don’t seriously address the risks these models can pose,” he added, referring to the AI company hacked by rogue OpenAI models earlier this year.

Adding to that challenge is that AI is not just a technology issue but a jobs, military, and whole-of-society issue, meaning there is no one congressional committee neatly situated to take the lead on wrestling with all its tentacles.

With China, a ‘prisoner’s dilemma’

Following Coxon’s resignation this week, Democrats and some Republicans were jolted out of their lethargy. They offered a flurry of concerned reactions and proposals to investigate AI companies and the pace of model advancement, including a proposal by Democrats to create a select committee on AI if the party retakes the House next year, Politico reported.

As the issue rises in importance in the upcoming midterm elections, former President Barack Obama warned at a private fundraising event last week that AI could be “dangerous,” and implored Democrats to develop “a very clear plan” on risks to safety and the economy.

Sen. Bernie Sanders (I., Vt.), who has for months been perhaps the loudest Cassandra on Capitol Hill warning about the societal risks of AI, invited lawmakers to a private briefing in the coming days to discuss concerns with AI safety experts.

Sanders has pushed for a ban on superintelligent AI, though other lawmakers have criticized the effort as stifling American innovation.

The congressional missives grew more urgent following the release of a report last Thursday by Anthropic that showed the many ways that actors across the globe, including in China, Russia, Iran, and Yemen, had been using its various Claude models to develop cyberattacks, surveil dissidents, and build missiles, bombs, and drones. The report also documented several cases of potential attempts to engineer biological weapons, which is considered among the most serious of existential threats related to AI.

One might have expected that report to stir action: It combined foreign belligerents and attacks on U.S. targets, though Anthropic blocked or stymied many of the documented cases of abuse.

Yet despite the alarm bells, lawmakers are far from agreeing on a way forward to regulate AI models or the companies that build them. Prospects for passing substantive legislation in the near term appear remote, in part because many see the issue as a “prisoner’s dilemma” created by the AI race between the United States and China. With AI dominance being the top national security issue of the century, both nations would doubtless be better off if they cooperated on limits. But they may not trust the other to abide by such an agreement. And in any event, unlike in nuclear arms control, compliance would be virtually unverifiable.

Sen. Ted Cruz (R., Texas), the chairperson of the Commerce Committee, acknowledged last week that he was deeply concerned about AI risks to humanity. But he also suggested that Coxon was part of a coordinated effort to support Democratic efforts to “strangle American AI,” and to hand leadership of the technology to China.

“I’d rather they be American killer robots and not Chinese killer robots,” Cruz told Politico.

Still, Cruz said he was working on legislation with Sens. Amy Klobuchar (D., Minn.) and John Thune (R., S.D.), the majority leader, aimed at addressing catastrophic AI risks related to biological and nuclear threats.

The likelihood of that or other legislative proposals — including requiring a “kill switch” in AI models that could immediately shut them down — gaining serious traction two months before an election appear slim. It is also not clear if a new Congress next year will be able to overcome sharp partisan divides to grapple with technology that is evolving so quickly, especially with Trump warning against regulation.

Some Trump administration officials have acknowledged that AI has dramatically increased the stakes. Nick Anderson, the acting director of the Cybersecurity and Infrastructure Security Agency, said that digital threats had become more urgent in the AI era and demanded quick action.

“We know the worst that can happen,” he said. Addressing the audience of government officials and cybersecurity executives at the Billington conference, Anderson offered a stark warning that if more was not done to protect the country, doomsday scenarios could become reality.

“You all are going to have to go home and look to your family, look to your friends,” he said, “and explain to them how you knew the worst that could happen and why we didn’t do enough.”

This article originally appeared in the New York Times.