For the past two years, much of the chatter surrounding AI security has largely centered around its output: Will it hallucinate and fabricate its own research? Leak sensitive data? Produce inappropriate or unsafe content that necessitates damage control later down the line?

Make no mistake, these all still remain valid concerns, and rightfully so. But as organizations move from chatbots to AI agents, the scope of risks have expanded beyond these issues. With AI agents not only generating answers, but also calling APIs, querying databases, triggering workflows, and interacting with entire business systems, the greater security risk is on permissions: what is the agent granted access to do—and more importantly, should it be allowed to do those things?
This is where most traditional AI guardrails fall short.

Guardrails Stop Outputs. Agents Take Actions.

When the concern is a faulty response, solutions like content moderation, prompt filtering, and output validation are all useful solutions. But AI agents introduce a different category of risk because they operate much closer to the action layer, where mistakes can have direct consequences on the business.

For example, unlike a chatbot that simply generates text, an AI agent can take actions on a user’s behalf. A support agent might update CRM records. An IT agent could interact with ServiceNow, Okta, or internal identity systems. A finance agent may retrieve ERP data or trigger approval workflows, while a DevOps agent might query observability platforms or initiate deployment processes.

To perform these tasks, agents need access to the systems that power the business: SaaS applications, identity providers, ERPs, ticketing platforms, data warehouses, internal APIs, and increasingly, MCP-style tool servers. During early pilots, organizations often grant broad permissions to accelerate experimentation and prove value quickly. While that approach may be acceptable in a controlled environment, it becomes far riskier at enterprise scale, where over-permissioned agents can create significant security, compliance, and operational challenges.

The problem is not that agents are malicious, but rather that they are software operating with delegated authority across multiple systems, often with incomplete context. A flawed instruction, prompt injection attack, misconfigured workflow, or runaway retry loop can quickly escalate a simple error from a “bad answer” into a “bad action”—one that modifies data, triggers a process, or exposes sensitive information.

Over-Permissioned Agents Create Real Business Risk.

Consider an IT operations agent connected to a ticketing platform, an identity provider, and a set of internal APIs. If it misinterprets a request or acts on a manipulated instruction, the consequences extend beyond an incorrect recommendation. The agent could reset the wrong user’s credentials, close the wrong incident, expose sensitive records, or repeatedly call a backend service until it begins affecting system availability.

The same pattern can play out in other parts of the business. For instance, a finance agent with broad ERP access could retrieve or modify sensitive payment data. Or a support agent could expose customer information. A DevOps agent could overwhelm production services with unnecessary requests, while a sales operations agent could modify CRM records based on incomplete or inaccurate context.

What makes these scenarios challenging is that they are not primarily model-quality issues. They are identity, authorization, governance, and operational control issues. The risk emerges when an agent is trusted to take action but lacks the guardrails needed to ensure those actions remain appropriate.

Most organizations already have mature controls for human users and traditional applications. They define roles, enforce least-privilege access, monitor activity, maintain audit trails, and require approvals for high-risk actions. Yet those same controls are often inconsistently applied when the actor is an AI agent interacting with tools and APIs on behalf of a user.
That inconsistency creates a new attack surface. As agents gain access to more systems and greater autonomy, the gap between what they can do and what they should be allowed to do becomes a growing source of enterprise risk.

Governance Has to Move to the Action Layer.

Addressing these risks begins with least-privilege access. Agents should only be granted the systems, tools, and actions required to perform a clearly defined task. But permissions alone are not enough. Effective access control must be contextual, taking into account who initiated the request, which workflow is being executed, what system is being accessed, and how sensitive the requested action may be.

Visibility is equally important. Organizations need a complete record of agent activity, from API calls and tool invocations to data access, workflow execution, and administrative changes. When something goes wrong, security teams should be able to quickly determine which agent performed the action, on whose behalf it acted, which systems were involved, what policies governed the decision, and whether the action was approved, denied, or escalated.

Beyond visibility, organizations need monitoring that is designed for agent behavior. An agent that suddenly accesses unfamiliar systems, generates an unusual volume of requests, or attempts actions outside its established patterns should raise immediate concerns. In this context, rate limits and usage controls are more than mechanisms for controlling cloud costs. They serve as important security and reliability safeguards, helping prevent runaway behavior from disrupting critical systems or business processes.

For the highest-risk actions, monitoring alone is not sufficient. Tasks such as moving funds, changing permissions, deleting records, deploying code, or modifying customer data warrant an additional layer of oversight. Human approval, step-up authorization, or other policy-driven checkpoints can provide that safeguard. The goal is not to insert friction into every workflow, but to ensure that consequential actions receive the level of scrutiny their potential impact demands.

Start With Ownership, Visibility, and Least Privilege.

Effective governance requires a clear understanding of who is responsible for what. Every sector of the enterprise, from security leaders to application owners to business stakeholders, must decide within their respective domains which actions agents can perform autonomously, and which require additional review. Without these responsibilities defined clearly, teams risk deploying agents faster than they can govern them, creating blind spots in oversight and accountability.
So where should organizations start? Inventory the agents already in use, map their access, and assign clear ownership. Once that foundation is in place, organizations can focus on the controls that matter most: limiting permissions, monitoring activity, and adding oversight for sensitive actions.

There is a world of potential and productivity that can be accessed through agents. Across IT, engineering, support, finance, and operations, agents are already helping teams move faster and automate work in unprecedented ways. But as agents become more capable, our expectations have to expand beyond model performance and prompt quality and move into the operational: What should an agent be allowed to do? Under what circumstances? And how does the organization maintain oversight when something goes wrong?

With business outcomes (and not just model outputs) at stake, governance is no longer a nice-to-have. It is the foundation for responsible AI adoption.