OpenAI’s AI models went rogue: They bypassed controls and hacked Hugging Face; Here’s what happened

OpenAI has officially revealed details about how its AI model went beyond its limits and breached the Hugging Face system. The company acknowledged that the model escaped the boundaries of its controlled test environment, accessed the internet and used multiple security flaws to breach another company’s system during an internal cybersecurity experiment.

OpenAI briefed on how the AI model independently combined persistence, tool use, and interaction with other models to bypass safeguards and cross the boundary of its test environment. However, the breach did not happen because of one specific security bug or failure, but because several factors led to the breach.

Must read: ChatGPT ads arrive in India as OpenAI targets Free, Go users in major revenue push ahead of IPO

How did OpenAI’s AI model go rogue?

In the company blog post, OpenAI revealed that it was testing an experimental AI model called Internal Model 1 (IM1)  in a sandbox, which is an isolated virtual computer. This is a safe environment to test AI agents and models, as they prevent the models from accessing the wider internet.

During the experiment, the model found a way to communicate with other models in many of these environments. OpenAI said that the models used an internal package-management system called “Artifactory” as an unofficial communication channel. This allowed the models to communicate and coordinate despite restrictions.

Must read: SoftBank could raise up to $20 billion in bonds to invest in OpenAI: Report

They also discovered ways to access the system’s internet connectivity to reach external websites. This is where the first breach occurred, when it bypassed the sandbox's direct internet restrictions and accessed resources beyond its intended environment.

How do AI models breach Hugging Face?

With greater capabilities and internet access, the models began investigating vulnerabilities outside their testing environment, which became a genuine security incident. The AI agents compromised Hugging Face systems by accessing credentials.

It chained together multiple vulnerabilities and accessed its infrastructure, including code execution and root-level access on at least one server. However, OpenAI assured that the incident did not compromise customer data or impact its products.

OpenAI called the incident episode as a “warning shot” for AI security.