AI tool poisoning exposes a major flaw in enterprise agent security
Artificial intelligence (AI) agents are revolutionizing the way tools are selected from shared registries by matching natural-language descriptions. However, a critical gap has been identified in this process – the lack of human verification to ensure the accuracy of these descriptions.
The discovery of this vulnerability came to light when a submission was made in the CoSAI secure-ai-tooling repository. Initially thought to be a single risk entry, it was later split into two separate issues by the repository maintainer. One issue addressed selection-time threats such as tool impersonation and metadata manipulation, while the other focused on execution-time threats like behavioral drift and runtime contract violation. This separation highlighted the fact that tool registry poisoning is not just one vulnerability, but a series of vulnerabilities that exist at every stage of a tool’s life cycle.
In response to this gap, there is a natural inclination to apply existing defenses such as code signing, software bill of materials (SBOMs), and supply-chain levels for software artifacts like SLSA and Sigstore. While these defense-in-depth techniques are valuable, they do not address the core issue of behavioral integrity – ensuring that a tool behaves as it claims and acts only as intended.
One attack pattern that artifact-integrity checks often miss is prompt-injection payloads, where a tool may manipulate the agent into always selecting it over alternatives. Another challenge is behavioral drift, where a tool may change its behavior after being verified, leading to potential security risks.
To address these issues, a runtime verification layer is proposed within the model context protocol (MCP). This verification proxy sits between the agent and the tool, conducting validations on each invocation to ensure discovery binding, endpoint allowlisting, and output schema validation. By incorporating a machine-readable behavioral specification as part of the tool’s signed attestation, this approach enables real-time verification of a tool’s behavior.
It is essential to note that neither provenance nor runtime verification alone is sufficient to address the complexities of tool registry security. A combined approach is needed to mitigate the risks effectively.
To implement this solution without hindering developer velocity, a phased rollout strategy is recommended. Starting with endpoint allowlisting at deployment time, organizations can gradually add output schema validation, discovery binding for high-risk tools, and full behavioral monitoring based on the level of assurance required.
In conclusion, while provenance-based controls like SLSA are valuable, they are only part of the solution. Organizations should prioritize implementing behavioral integrity measures to ensure the security and integrity of their AI agent tool registries. By taking proactive steps to enhance tool registry security, companies can mitigate the risks associated with malicious tool behavior and safeguard their AI platforms effectively.
