BT explainer | Machine vs. man: Leaders building AI think it could kill us. But should we be afraid?

Will artificial intelligence (AI) kill humanity in the coming years? Several AI researchers and tech leaders have come forward, expressing concerns about its rapid advancement and the potential risks it could pose to humanity. But what caught the attention? Well, it was not a new AI product launch, not a funding round, not a benchmark, but a simple resignation letter.

Jacob Coxon, a 27-year-old researcher who had worked at both OpenAI and Anthropic, posted on X that he was quitting the AI industry. "Neither company is acting responsibly," he wrote. "They are racing straight to self-improving superintelligence and gambling with our lives." The post gained massive traction, leading people to debate about pacing advancements in AI.

Must read: Anthropic’s AI researcher exits AI industry with a warning: ‘Do not underestimate its power’

By the weekend, the CEOs of the three most powerful AI companies in the world, Dario Amodei of Anthropic, Sam Altman of OpenAI, and Elon Musk of xAI, publicly agreed that AI development must be slowed down. But what’s causing the sudden fear, and why are tech leaders and government representative are having an elongated debate over AI, its risks, its development, and the need for greater regulation?

What is superintelligence? Why are researchers so afraid?

Superintelligence is a term used in the tech industry for an AI system that outperforms human capabilities. However, it does not yet exist. But the trajectory of AI development over the years has made the claims less hypothetical and more real.

Coxon in the post revealed that people building or working on AI privately raised concerns about the technology, saying that it could soon become catastrophically dangerous. "I spent the last three years doing pretraining research at both OpenAI and Anthropic," he wrote. "Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives."

Van Hubinger, Anthropic's head of alignment science, agreed with claims saying, “We really do earnestly believe AI could kill all humans.” He also estimated a greater than 10% probability of human extinction from misaligned AI within the decade.

Must read: 'Don't kill the Golden Goose': Trump says US must keep its AI lead over China

Well, this is not the first time an AI researcher has come forward. Previously, in February 2026, Anthropic's AI safety lead, Mrinank Sharma, also resigned for similar reasons.

However, a recent incident of AI models going rogue and conducting unauthorized cybersecurity attacks against targets outside their assigned parameters fueled the debate. In a detailed report, OpenAI revealed that its AI model breached an AI startup, Hugging Face, acting autonomously and exploiting vulnerabilities beyond the scope of its assigned task. On the other hand, Anthropic revealed a similar incident, creating concerns about AI working autonomously despite having set guidelines. Furthermore, the company also identified over five cases between December 2025 and August 2026, involving people using Claude AI models to support biological weapons development, although the research did not find enough evidence to prove that these people intended to use the information to carry out an attack.

What are AI developers saying about backlash?

On September 13, Anthropic CEO Dario Amodei published an essay about how the pace of AI advancement must change. He warned that AI agents could take over the internet within six months and emphasised stronger safeguards. To slow down the pace, he laid out a three-step plan for AI companies and governments to keep capable models aligned with human values.

Both Elon Musk and Sam Altman publicly endorsed Amodei's position. "Dario is right," Musk posted on X. The call for a deliberate AI slowdown has created significant concern among investors. Many have also started selling shares, pushing stock prices down. This comes at a time when both OpenAI and Anthropic are preparing for an IPO.

Must read: Trump calls data centre backlash a ‘hoax’ in call with Nvidia’s Jensen Huang; rules out AI slowdown debate

On the other hand, Microsoft CEO Satya Nadella also joined the conversation, expressing his take on the current AI development. “Any pursuit of superintelligence has to be grounded in the core principle that if the AI we build is not helping humanity and under human control, it's not worth pursuing,” Nadella said. He also supported the idea of both closed- and open-source model development and urged research and a deliberate pace.

Now, the debate revolves around why they are not stopping. Amid confessions about the potential AI risks, Nvidia CEO Jensen Huang says that an AI slowdown will not happen, building on US President Donald Trump’s stance that the rapid development of AI should continue.

Trump vs. AI slowdown: What governments say

In recent public interactions, Trump has publicly dismissed fears of AI. During his recent visit to Ireland, Trump said, “We're leading China on AI... and, frankly, I want to keep it that way because whoever wins AI wins.”

In a recent call with Nvidia’s Huang, Trump called fears a "hoax" and a "sick conspiracy," arguing that greater regulations are unnecessary and that slowing down AI development would only benefit global competitors like China.

Vice President J.D. Vance also supported Trump, saying the future of AI "is not going to be won by hand-wringing about safety.” On the other hand, David Sacks, Co-chair of the President's Council of Advisors on Science and Technology, argued that if unreleased models are dangerous, then tech companies must slow down themselves rather than ask for government intervention.

Must read: AI safety warning: Sam Altman, Elon Musk back Anthropic CEO’s call to slow down frontier AI. Why?

Could AI actually ‘kill’ us?

As per Anthropic’s Hubinger, the possibility of AI killing humans is greater than 10% within the next decade. Geoffrey Hinton, also known as the godfather of modern deep learning, has put the risk of AI causing human extinction at 10 to 20%. A Cambridge review also revealed that AI could accelerate technologies like bioscience faster than human understanding can keep up, hinting at the potential loss of human function and autonomy.

However, the 2026 International AI Safety report highlighted that current AI systems are closer but far from showing capabilities at levels that could trigger loss of control. However, the risk is real as per experts, but it may not come today, not this year, and possibly not this decade. However, considering the pace at which AI is progressing, the risks are certain if not contained.

Therefore, the honest answer to whether they are right is: nobody knows. And that uncertainty, more than any specific capability or any specific risk, is what last week's debate was really about.