AI safety alarms grow: ‘Tens of thousands’ of incidents reported across companies— some could be criminal

AI companies have reportedly faced a large number of cases where AI models behave outside their restrictive environment and perform beyond their intended limits during testing. According to an Axios report, researchers have found thousands of cases where AI systems behaved in ways they were not supposed to, and some autonomous AI behaviour could potentially amount to criminal activity.

It was highlighted that some AI systems even went beyond generating problematic responses and took unusual actions within digital environments, such as interfering with or taking control of other systems. This has raised concerns about how AI is shifting from simply producing unsafe content to AI agents independently taking actions they weren't authorised to take.

Must read: AI out of control: US govt websites hit as OpenAI bots bypass security controls

OpenAI, Anthropic AI agent went rogue

OpenAI agents allegedly leaked 53 images from ChatGPT users online, and it was also reported that the AI agent breached an Australian government website and other websites, including sites belonging to the US government. As a result, OpenAI has stopped training its frontier models and will resume only when safeguards are aligned.

CEO Sam Altman shared a X post saying, “We have not been as fast as we would have liked, but we are trying to balance our desire for transparency with gaining a clear understanding from petabytes of agent activity logs, and working with impacted organizations.” However, amid AI agents going rogue,

Altman highlighted that the Hugging Face incident was “the most severe event we’ve seen.” After the incident, several AI companies, including OpenAI, Anthropic, and Google, have urged an AI slowdown.

Must read: OpenAI AI agent bypassed restrictions on Australian government health website while searching for government data

On the other hand, Anthropic is also testing its AI models for misalignment, and it has hired an independent safety organisation to examine this behaviour. In one incident, Anthropic revealed that in the Claude Opus 5.5 system card, the model tried to escape its sandbox in 1.5% of test runs. Anthropic's Mythos model reportedly attempted this in 25% of tests.

As of now, it is reported that researchers are still evaluating each incident, and not all of the reported AI safety incidents were actual real-world failures, as some were deliberate attempts to see whether it breaks its safety rules. The purpose is to find weaknesses before real users or attackers discover them. Furthermore, the majority of these incidents haven't been made public yet, so the full scale and nature of the incidents aren't known.