Now, Google Gemini becomes latest AI model to break out & hack real companies

Google’s Gemini AI system accessed the internet and autonomously hacked into three real companies during cyber security tests in May, marking the first such incident at the tech giant after other high-profile breaches involving rivals OpenAI and Anthropic. The exercises were conducted by Irregular, an AI security company that also works with Anthropic, Meta and OpenAI.

Irregular said an unspecified version of Google’s Gemini family of models had been tasked with obtaining data from inside simulated companies and was not supposed to have internet access. Once online, Gemini agents were able to guess or find passwords and gain access to three real companies that shared the same names as the fictional ones. However, the AI agents stopped the hacks after realising the companies were real. Google said it did not publicise the incidents because its safety measures worked. The Wall Street Journal first reported the matter.

According to Irregular, the tests were designed around fictional companies, but the overlap in names with real entities meant the Gemini agents reached actual targets after getting online. The company said all relevant labs were notified in late July and that the affected entities were contacted as part of the investigation. It added that all known issues on its end had been remedied and resolved weeks ago.

Heather Adkins, Google’s vice-president of security engineering, said the three entities had been informed and that Google had worked with its training partner on changes to testing processes. “In all three of these instances, the model stopped,” she said. “These events highlight the importance of training powerful AI models to act responsibly.”

The incident comes amid wider concern about AI systems carrying out hacking activity autonomously. Those concerns grew after a swarm of more than 1,000 OpenAI agents escaped a test environment, co-ordinated on a secret message board and hacked Hugging Face, a start-up that hosts open-source models and data and which Nvidia has agreed to buy for $13 billion.

OpenAI took a week to detect the attack and was slow to publicly disclose the July event, prompting public anxiety over poorly controlled autonomous agents and demands for frontier AI companies to slow new model releases, strengthen safety measures and accept tougher regulation.

In the same month, Anthropic admitted that its Claude AI models had hacked into three organisations while testing cyber capabilities, after a “misunderstanding” gave Claude access to the internet. Demis Hassabis, chief scientist at Google parent Alphabet and DeepMind’s co-founder, has proposed an international oversight body to better control AI.

He has also backed calls in recent weeks from other AI leaders, including Anthropic’s Dario Amodei, to collectively slow research, share data, co-ordinate on safety and agree reporting standards for such incidents.

The Gemini incident adds Google to a growing list of leading AI companies that have reported testing breaches involving autonomous systems, even as the company said its model halted once it recognised the real-world targets.