'AI models cannot be trusted': Microsoft warns of insider threats, expands Project Perception

Microsoft has warned that powerful artificial intelligence models cannot be automatically trusted, even when deployed within companies. The warning comes as AI agents gain the ability to make decisions, access sensitive information and take actions without direct human intervention.

Microsoft CEO Satya Nadella on Monday announced an expansion of Project Perception, the company's AI security initiative, saying organisations must treat AI models as potential insider risks rather than relying on the models themselves to behave safely.

"I recently wrote about why we need to treat models as insider risks, with security controls sitting outside the model itself," Nadella wrote. "Today, we're putting that principle into practice by expanding Project Perception with new capabilities to help continuously assess agent risk, adapt permissions, and contain threats."

'Frontier models cannot be trusted'

In a blog post shared by Nadella, Hayete Gallot, Executive Vice President of Microsoft Security, warned that companies have been too quick to trust AI agents simply because they deployed them internally.

"Unfortunately, frontier models, regardless of their origin, cannot be trusted," Gallot wrote. "This is not because models are inherently malicious. Models have been trained to attain results, which often leads to reward hacking, and models are not optimized around organizational norms or judgment."

According to Gallot, AI agents can adapt to changing situations, behave unexpectedly, and even collaborate or collude with other agents to achieve their objectives without being compromised by hackers.

"Agents are unpredictable. They will adapt as the context changes, may not behave as intended and will collaborate and collude with other agents to reach their objectives, without needing to be compromised," she wrote.

The warning comes as businesses increasingly deploy AI agents to handle tasks involving company data, software tools and critical business operations.

Why Microsoft calls AI agents 'insiders'

Unlike conventional AI chatbots that primarily respond to user queries, autonomous AI agents can perform tasks, access applications, use tools and make decisions.

Gallot warned that many enterprise AI agents have excessive permissions, giving them access to systems and information beyond what they may need for their assigned tasks.

"Agents have been unleashed in the enterprise, at scale, and their numbers are growing rapidly. These agents are frequently overprivileged, have access to data, tools and systems and can take actions and make decisions in critical processes. They have become active participants in the enterprise," Gallot said. "They are agentic insiders."

She argued that companies need security systems capable of continuously monitoring what AI agents are doing and restricting their actions when risks emerge.

To address these concerns, Microsoft is expanding Project Perception, which was announced in July as an AI-driven cybersecurity initiative.

The system uses specialised AI agents to help security teams detect and respond to threats.

Under the expanded approach, Microsoft proposes using what the industry calls Guardian Agents to supervise other AI agents before they take actions.

These systems would monitor activity around the clock, assess risks in real time and adjust permissions when necessary.

Project Perception would bring together information from identity systems, cloud infrastructure, networks, endpoints and other security layers to identify agents operating within an organisation.

It would then assign identities, permissions and security policies to those agents.

As an agent's risk profile changes, Guardian Agents would be able to intervene and restrict its activities.

Microsoft describes this approach as applying predictable security controls to AI systems whose behaviour is not always predictable.

Microsoft seeks greater visibility into AI reasoning

Gallot also argued that monitoring an AI agent's actions is not enough. Security teams increasingly need to understand why an agent is taking a particular action and whether it matches the original task.

"Today, security systems can reliably observe the actions of the agents. Those signals are essential and tell us what we need to contain. Our security controls should increasingly be able to ask not only 'What is happening?' but also 'Why is it happening?' Is this consistent with the intent of the task?"

This will require greater transparency into the model itself, not only the input and output tokens, but the thinking and chain of thought tokens, she added.

"Only then can we truly understand whether the model's intent is aligned with the purpose of the agent. Model transparency will not replace containment. It will make containment smarter and stronger. And it will allow us to better inform accountability and traceability."

Microsoft's warning comes amid growing concerns among AI industry leaders over the pace of development.

In September, Anthropic CEO Dario Amodei called for slowing the development of advanced AI models to allow stronger safety checks, a proposal backed by OpenAI CEO Sam Altman, who agreed that the industry needed to "pace the frontier".