AI agents are becoming an enterprise architecture challenge

AI agents are becoming an enterprise architecture challenge

AI agents are starting to act on their own inside business systems and governance frameworks aren’t ready for it. Kumaran C R, Vice President & Managing Director ANZ, UST, says organisations need to incorporate controls into their systems from day one.

The cyber incident involving an AI agent exploiting a vulnerability in a gym booking system serves as an early warning for enterprises moving from AI experimentation to deployment.

In the incident, an AI assistant was asked to book a gym class but found a way around the booking system’s limits and removed another person from a waiting list – actions its user had not asked it to take.

The significance of the incident is that the agent did not need to be malicious to create risk. It simply identified an unintended way to achieve its objective, exposing a new category of governance challenge as AI systems move from generating outputs to taking autonomous action.

As organisations begin embedding agents into business processes, governance can no longer sit as a policy or security layer applied after deployment. It needs to be designed into the architecture around agents from the outset, including the data they can access, the tools they can use, the workflows they can influence and the controls that keep their actions aligned with business intent.

AI governance is entering a new phase

Over the last few years, much of the AI governance conversation has focused on how AI generates content, recommendations and insights. Questions around bias, hallucinations, transparency and responsible AI remain important, but they were largely developed in a world where humans remained responsible for taking action.

Agentic AI changes that equation.

The gym booking incident illustrates the shift. The risk did not arise from malicious intent, but from an autonomous system pursuing a legitimate objective in an unintended way.

Traditional governance approaches are often designed around preventing misuse or managing the outputs of AI systems. Agentic AI requires organisations to ensure autonomous systems remain aligned with organisational intent when they encounter ambiguity, incomplete information or unexpected conditions.

That challenge is becoming more immediate as organisations begin deploying agents across business processes. The Governance Institute of Australia estimates that just over half of organisations have deployed AI agents, bringing questions of authority, accountability and human oversight into sharper focus.

As agents connect to business processes and operational workflows, governance needs to extend beyond what the model produces to what autonomous systems can access, influence and do.

Governance extends beyond the model

Many organisations still think about AI governance primarily in terms of model risk, but in practice, the greatest risks often sit in the environment around the model.

Consider a customer service agent connected to a CRM platform, knowledge base and workflow automation system. The model may be functioning exactly as intended. Yet the outcomes it produces are shaped by the broader environment in which it operates, including the data it can access, the tools it can use and the workflows it can influence.

This includes the prompts an agent receives, the APIs it can call, the memory it retains and the surrounding systems it can influence.

In an agentic environment, organisations are no longer governing a model in isolation. They are governing a network of systems, permissions, data sources and workflows that together enable autonomous action.

Governance needs to operate in real time

As AI agents become embedded in business operations, organisations need to think beyond model performance and static pre-deployment review.

Traditional testing focuses on whether a system can complete a task successfully. Autonomous systems introduce a broader challenge because they continue operating in changing environments, interacting with evolving data, workflows and enterprise systems.

This is why testing for failure becomes just as important as testing for performance. Organisations need to understand how autonomous systems behave when assumptions break down and whether the surrounding controls can maintain trust in the outcomes they produce.

In many cases, the weak point is not the technology itself, but the surrounding systems, controls and operating processes that were never designed to support autonomous decision-making at scale.

That requires governance to become an operational capability rather than a compliance exercise. Organisations need visibility into how agents behave, what actions they take and how decisions are made. They also need mechanisms to intervene when behaviour falls outside expected parameters and audit trails that support accountability.

An operating-model challenge

The more deeply AI is integrated into enterprise operations, the more governance becomes an organisational challenge rather than a purely technical one.

The issue is integrating AI capability into the organisation in a way that remains transparent, accountable and aligned with business objectives.

As autonomous systems take on greater responsibility, organisations need clear ownership of how those systems are deployed, monitored and managed. They also need defined intervention points, appropriate human oversight and mechanisms for recording and auditing decisions.

Governing autonomous systems ultimately requires organisations to rethink not only their technology architecture, but also the operating models that sit around it.

Governance needs to be built in, not bolted on

The organisations that succeed with Agentic AI will not necessarily be those with the most advanced models. They will be those that treat governance as part of enterprise architecture from the outset.

That means designing for visibility, containment, accountability and intervention before agents are deployed at scale, so governance can evolve with the technology rather than lag behind it.

As AI agents become more deeply embedded in enterprise operations, governance is moving from policy to production.

And that shift is turning AI governance into one of the most important enterprise architecture challenges organisations will face in the years ahead.

Browse our latest issue

Intelligent CIO APAC

View Magazine Archive