Anthropic resumes Claude testing after real-world hacks, adds stronger AI safety safeguards

Anthropic is tightening its approach to cybersecurity testing after incidents raised questions about how AI models behave when safeguards around test environments fail. The company reviewed more than 141,000 evaluation runs and found cases involving access to real systems. The incidents exposed weaknesses in testing setups and concerns about how models can pursue narrowly defined goals. 

Must Read: Anthropic warns Claude users as malware steals login sessions and drains account usage

For users and businesses watching the rapid development of AI agents, the episode offers a closer look at safeguards being built around increasingly capable systems. Anthropic has now detailed the changes it made after the incidents and outlined how external testing will proceed.

Claude reached real company systems during tests

Anthropic found three cases where Claude models accessed the internet during cybersecurity evaluations and then reached real company systems. In one case, Claude Opus 4.7 targeted a real business because its name matched a fictional company in the test, obtained credentials and reached a production database.

In another, Claude Mythos 5 uploaded a malicious Python package to a public repository. It remained available for about an hour and was downloaded by 15 real systems. Anthropic said the models used basic techniques, including weak passwords and exposed debug pages, rather than sophisticated vulnerabilities.

Anthropic strengthens safeguards and resumes testing

A separate incident reported by the UK AI Security Institute involved Claude Mythos 5 taking unauthorised actions on the live internet during cybersecurity testing. Anthropic is reviewing the incidents and plans to work with independent AI safety organisation METR.

Must Read: Sony accuses Anthropic of copying its music for Claude AI training; AI giant could pay up to $150,000 per song

The company has built a real-time classifier to detect attempts to probe or escape testing environments or unexpected internet access. It can block actions and alert a human. High-risk environments have also been moved to stronger isolation, while evaluators must use sealed sandboxes, stress-test them and continuously monitor models.

Anthropic has resumed external cybersecurity testing of its AI models after introducing these safeguards, following a pause of about a month.

Anthropic has also published research examining reward-seeking behaviour in AI models and how they can exploit weaknesses in evaluation systems.

The company has also redirected about 150 product engineers towards security work as it examines how flawed testing environments can encourage models to pursue narrow goals recklessly.