'Strong chance that we could all die': Ex-Anthropic AI researcher warns rapid progress could threaten humanity

Jacob Coxon, an AI researcher who recently left Anthropic, has warned that the rapid pace of AI development could have dangerous consequences, saying people working in the industry are "genuinely frightened" about where the technology is heading, the BBC reported on Sunday.

Coxon, 27, who previously worked at OpenAI, said he believed the risks could become severe if the current pace of development continued.

"I believe that if we don't slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future," he said in an interview with the BBC's Laura Kuenssberg.

AI workers ‘trapped in a race’

Coxon's resignation comes as concerns about AI safety grow within the industry. His former boss, Anthropic chief Dario Amodei, has argued that AI development should slow down, saying the technology itself was not in question but its risks were "serious".

OpenAI chief Sam Altman and Elon Musk have also said they agree with Amodei's proposal for industry-wide deceleration and regulation, as well as independent monitoring of AI models as they are developed.

Coxon welcomed a slowdown but said it would need to be coordinated with China to prevent an international AI race.

"The people who work at these companies are completely serious when they ask for regulation because they find themselves trapped in a race. And they're scared of the outcomes of that race," he said.

Don't Miss: Nvidia's Jensen Huang declares 'AGI has arrived' as OpenAI unveils GPT-6 Astra

Could AI take over the internet?

Coxon said one of the hardest questions was what an AI apocalypse could actually look like.

Amodei has warned about a possible swarm of bots acting like a supercomputer and taking over the internet. Coxon said such a scenario could become realistic within six months to a year.

He said some AI workers were already thinking about the possible consequences of rapid advances, with some "planning what to do with their lives and thinking about the impacts of their work", while others were "considering buying land somewhere because they're so scared of the instability as a result of rapid AI progress".

"They all keep this in their head on a daily basis while working on the technology."

Coxon said some of his peers feared the danger could arrive within the next two years. He nevertheless said AI researchers "genuinely want to see the upside" of the technology, including "solving diseases and improving everyone's lives".

Anthropic defends AI safety work

Anthropic said it had always been transparent about both the benefits and risks of AI.

"We have always been transparent that AI will bring both enormous benefits and unprecedented risks.

"To address these risks, we continue to build models with some of the strongest safeguards in the industry."

The company said it "aggressively" tests its models and publishes findings to aid scrutiny and prevent incidents of "AI misalignment". It also said the industry would benefit from a "lawful, verifiable way" to work together on the pace at which powerful models are released.

Silicon Valley pushes back

Coxon's warnings have also faced criticism.

Hugging Face CEO Clement Delangue wrote on X: "Sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change." He later offered to help with Amodei's proposed solutions.

Nvidia chief Jensen Huang has also dismissed similar warnings. Multiple people at a Goldman Sachs conference told the BBC that Huang rejected Coxon's comments as untrue. Huang has previously called the idea that AI was "going to be the end of humanity" "complete nonsense".

Anthropic scientist Evan Hubinger, however, agreed with Coxon.

"We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade," Hubinger said.

The debate comes as AI models become more capable. Anthropic said in April that it was withholding its Mythos model because it could independently escape its testing environment, known as a sandbox. OpenAI has also said some aspects of its Astra model's development were paused over cybersecurity concerns.