Tag Archives: Access Control

Security Risks of Agentic AI: Protecting Enterprise Systems and Integrations

AI Security

An AI assistant that summarizes a document creates a different security exposure from an agent that can search company systems, send messages, or modify business records. Agentic systems combine model-generated decisions with real permissions and external tools. When the model misinterprets information—or an attacker manipulates it—the consequences can extend beyond an incorrect answer to unauthorized actions.

OWASP’s Top 10 for Agentic Applications highlights risks such as goal hijacking, tool misuse, and identity or privilege abuse. NIST has also evaluated agent hijacking through malicious instructions embedded in information an agent reads. These risks do not make enterprise agents inherently unsuitable. They show why security controls must govern the complete workflow, not merely the model’s instructions.

Why Agentic AI Changes the Threat Model

Traditional application security assumes developers specify the operations a system performs. An agent introduces a decision-making layer that may choose among tools, assemble arguments, and sequence actions based on natural-language requests and retrieved content. The tools may be conventional APIs, but the choice to call them may depend on model output.

That expands the attack surface across prompts, retrieved documents, tool descriptions, permissions, integrations, and interactions between agents. A secure design must distinguish trusted instructions from lower-trust data and enforce authorization outside the model. Telling an agent to follow company policy is not a substitute for validating the actions it attempts.

Prompt Injection and Agent Goal Hijacking

Indirect prompt injection occurs when attacker-controlled content attempts to instruct an agent that is supposed to treat that content as data. A malicious document, support ticket, search result, or tool response might claim to supersede the user’s request, demand a new action, or direct information to an outside destination. If the agent follows it, the attacker can redirect the workflow without controlling the original user.

Consider a service agent reviewing a customer attachment. The attachment contains legitimate order information but also hidden instructions to email account records to an external address. The application should treat every instruction inside that attachment as untrusted, even if the text resembles a policy update or administrative command.

NIST’s research on agent hijacking describes this general failure mode: malicious instructions inserted into data can steer agents toward unintended actions. Defensive measures include maintaining trust boundaries between instructions and retrieved content, restricting permitted tools and destinations, and evaluating attacks against the complete agent workflow. No prompt-only technique reliably eliminates the risk.

Excessive Permissions and Identity Confusion

An agent can do only what its connected services allow, so permission design determines the consequences of a compromised decision. A customer-service assistant may need order status but not access to payroll records, payment credentials, or administrative settings. Shared service accounts with broad access can turn one misleading input into a cross-system incident.

Organizations should distinguish the employee requesting an action from the identity under which the agent executes it. Some integrations act on behalf of an end user; others use an application or service identity. Each needs clearly scoped privileges, credential rotation, and authorization checks at the destination system. Sensitive operations should require approval tied to the actual transaction, not a blanket approval given when an integration is installed.

The Model Context Protocol’s authorization guidance addresses token audience restrictions and warns against token passthrough that can enable confused-deputy problems. A tool server should not accept credentials meant for another service or silently forward broad authority downstream. Protocol compatibility alone does not establish correct authorization.

Tool Poisoning and Untrusted Integrations

Corrupted Code

Agents may select tools based partly on names, descriptions, and parameter schemas exposed by connected servers. If a malicious integration supplies deceptive metadata, it may influence which tool an agent chooses or how other tools are used. OWASP’s MCP Security Cheat Sheet describes tool poisoning, tool shadowing, and changes to previously trusted tool definitions as important risks.

Treat tool servers as software dependencies with their own supply-chain exposure. Review the publisher, package provenance, update process, network access, and data permissions before deployment. Pin or review important changes where possible, isolate execution environments, and monitor tool catalogs for unexpected modifications. A description that says a tool is safe cannot serve as proof that its implementation is safe.

Data Leakage Through Legitimate Actions

Agents can leak sensitive information without invoking an obviously malicious tool. An attacker might persuade an agent to include internal data in an outbound email, a ticket comment, a web request, or a seemingly routine search query. Ordinary integrations can become exfiltration channels when the agent’s decision process is manipulated.

Limit which sources an agent can retrieve and which destinations it may contact. Apply data classification and output checks to high-risk transfers, suppress secrets from logs and prompts, and require explicit confirmation before sharing sensitive information outside an approved boundary. Retrieval systems should enforce document-level permissions rather than assuming that everything in an index is safe to disclose.

Unsafe Actions, Duplicates, and Cascading Failures

A harmful result is not always caused by an adversary. An agent may retry a tool call after a timeout, misread a success response, or use outdated records when making a decision. Repeating an inventory adjustment or payment request can damage operations even when the original task was legitimate.

Every consequential action needs validated inputs, defined spending or change limits, and confirmation from the system of record. Use idempotency mechanisms where available to prevent duplicate effects, record the state before and after important transactions, and provide safe recovery from partial failures. Human approval should cover the actual proposed action and its consequences.

In multi-agent workflows, an error can spread when one component passes an unsupported conclusion to another. Each handoff should include provenance, expected outputs, and error conditions. Separate agents may have different permissions, but that separation works only when enforced by underlying systems rather than by descriptions of their roles.

An Enterprise Example: A Compromised Procurement Workflow

Imagine a purchasing agent that reviews vendor emails, checks purchase orders in an ERP system, and prepares requests for approval. An attacker sends a document that looks like an updated supplier instruction but includes directions to change a payment destination. The agent summarizes the document and proposes an ERP update.

A weak implementation might treat the document as authorized guidance and attempt the change using an overly privileged account. A stronger design verifies supplier changes against authoritative records, restricts the agent to drafting rather than approving payment updates, and routes the proposed change through an independent verification process. The final ERP action is executed only after appropriate approval and validation.

This scenario illustrates why source trust, privileges, and transaction safeguards must work together. Detecting malicious wording helps, but access boundaries and approval controls limit damage even when detection fails.

Testing and Monitoring Agent Security

Agent security testing should include malicious documents, poisoned search results, altered tool metadata, unexpected API responses, expired credentials, unauthorized requests, duplicated operations, and attempts to move information across permission boundaries. Tests should cover complete sequences of retrieval, tool selection, execution, and verification—not just isolated model responses.

Record which identity initiated an action, which tool executed it, what source data informed it, and the resulting state. Watch for unusual destinations, repeated failures, expanding permission requests, and unexplained changes in tool usage. Security teams also need a way to pause an agent, revoke its credentials, reconcile unfinished transactions, and return the workflow to manual operation.

Measure prevented unauthorized actions, accurate escalation, false positives, recovery time, and the cost of human review alongside productivity gains. NIST’s AI Risk Management Framework offers a structure for governance and ongoing assessment, but it is not a substitute for product-specific penetration testing and operational security controls.

Building Security Into Agentic AI From the Start

The safest starting point is a narrowly scoped agent with read-only access, clear business ownership, and a well-defined outcome. Add write permissions incrementally, only after the workflow has been tested against adversarial inputs and ordinary integration failures. Prefer deterministic application rules for irreversible actions and preserve meaningful human checkpoints.

Enterprise agent security ultimately depends on controlling authority, not trusting generated intentions. Strong identity management, trustworthy integrations, bounded tool use, transaction verification, and incident response make it possible to benefit from agentic automation without giving a model unrestricted control over business systems.