Anthropic’s AI researcher exits AI industry with a warning: ‘Do not underestimate its power’

Anthropic’s AI researcher, Jacob Coxon, announced that he is quitting the AI industry amid concerns over progression, security, and control. He revealed that the ongoing AI race has started to become a major security problem and showed concerns over companies such as Anthropic and OpenAI moving too quickly to develop extremely powerful AI systems.

Coxon, an AI researcher who specializes in training new AI models, revealed that he has worked at both OpenAI and Anthropic. In a detailed X post, he revealed why he is quitting the AI industry and his fears surrounding the rapid AI progress.

Must read: Nvidia's Jensen Huang declares 'AGI has arrived' as OpenAI unveils GPT-6 Astra

“I resigned from Anthropic today,” he wrote in the post. He revealed that companies like Anthropic and OpenAI are “Tracing straight to self-improving superintelligence and gambling with our lives.”

AI will soon become “superhuman systems”

In the thread X post, Coxon said: “Do not underestimate the power of this technology.” Talking about the rapid progress, he revealed that AI can “revolutionize any field overnight and acquire real power and resources.”

He revealed that his concerns stem from conversations within the AI industry, where executives and researchers acknowledge the possibility of catastrophic outcomes privately. “The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon said.

“I hear the same people express fear privately. No other human activity poses this level of danger,” he added.

AI race continues despite fears

The researcher further revealed why AI companies continue building increasingly powerful AI despite knowing the potential risks. According to Coxon, OpenAI has not fully grasped the seriousness of the potential consequences. He says the company does "not deeply internalize the civilizational stakes.”

Must read: ‘No one is prepared for the consequences’: OpenAI chief scientist calls for ‘extreme caution’ over AI’s rapid progress

Whereas Anthropic's researchers understand the potential risks, but they are caught in an AI arms race. He revealed that Anthropic believes that “no one else will act responsibly, so they must do it themselves, despite the risk.” Coxon calls the logic “hubristic gamble,” a bet that humans can safely develop extremely powerful AI before losing control of it.

He also urged researchers to speak up and demand corrective safety measures, more oversight, and stronger guarantees before developing increasingly powerful AI systems. "I don't feel like we're on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities."

Anthropic responds to Coxon's claims

Evan Hubinger, Alignment Science Lead at Anthropic, backed Coxon's claims, saying that “Jacob is correct here-we really do earnestly believe AI could kill all humans!”

“I personally think it is >10 per cent within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

Hubinger further said that he does not believe current AI systems pose a major existential threat to humanity. His concern is about a future AI that can improve its own capabilities repeatedly.