AI agent tools aren’t ready for the real world yet, new study finds

Software frameworks that power AI agents, systems designed to carry out tasks on their own, are improving fast, but most are still too immature for large-scale, real-world use, according to a new study.

The research paper, Agentic AI Frameworks Under The Microscope, was authored by Karthik Vaidhyanathan of International Institute of Information Technology Hyderabad and Davide Taibi, chair of software architecture at the University of Southern Denmark. It was published in IEEE Software, a peer-reviewed magazine and scientific journal.

The paper examines today’s most popular “agentic AI frameworks”, software platforms that let developers build AI systems capable of planning work, using tools and collaborating with other AI agents. These frameworks are increasingly used to automate tasks such as writing code, managing emails, analysing data or coordinating workflows.

However, while interest is booming, the underlying technology remains fragile.

Also read: Your assistant, your machine, your risk: Inside OpenClaw's security challenge

“Most agentic AI frameworks are still immature, with limited memory, basic planning, and uneven ecosystem support,” the paper says, warning that companies should be cautious because “scalability and reliability remain open challenges.”

What works and what doesn’t

One area of progress is modularity. In simple terms, most modern frameworks now allow developers to plug in external tools such as databases, APIs or business software. This makes it easier to connect AI agents to real systems.

Where they fall short is memory.

Many platforms rely on something called vector storage, a technique that saves information as numerical representations so AI models can retrieve past context. While this helps agents “remember” earlier conversations or tasks, it is still basic and short-lived.

 

Some frameworks share this memory across multiple agents, allowing them to collaborate. Others only keep information during a single session. That makes them easy to test, but not suitable for long projects that require continuity over days or weeks.

Only a few systems support what the researchers call global state, a central memory that coordinates everything happening across agents. 

Also read: How India’s IT sector is facing up to the AI threat

The research paper notes that planning is another weak spot. Today’s agents mostly use simple trial-and-error or reactive approaches, trying different paths until something works, rather than deeper reasoning strategies that combine logic, rules and learning.

Big communities, uneven foundations

The study also looked at developer adoption. Several open-source projects have attracted tens of thousands of contributors and followers, while newer frameworks are gaining popularity quickly.

 

Most tools are built primarily in Python, the dominant language for AI development. Some also support JavaScript or enterprise languages such as C# and Java, reflecting attempts to reach both startups and large companies.

Also read: Era of AI-led experimentation is over: Coforge's Sudhir Singh

However, key enterprise features are missing or incomplete:

• Observability (the ability to monitor what AI agents are doing and why) is inconsistent. Some frameworks offer detailed tracking, while others provide little more than basic logs.
   
• Security is underdeveloped, usually limited to simple safeguards like sandboxed execution or policy checks.
   
• Interoperability, letting different agents and tools talk to each other, is improving through shared standards, but remains early.

A long road to production use

The authors argue that today’s agentic AI platforms are closer to experimental infrastructure than finished products.

To move beyond prototypes, the paper calls for major improvements in persistent memory, hybrid reasoning (combining multiple thinking methods), stronger security and better monitoring.