← Back to perspectives
/Intellicon FDE Team

Why Can a Tool That Worked Normally at Installation Later Perform Malicious Actions?

Provider-supplied tool descriptions can cause an AI Agent to change its original execution plan. After integration, businesses need to keep reviewing changes, restrict outbound connections, and establish rules for writing to memory and handling abnormal behavior.

AIEgentWrX導入人機協作
Why Can a Tool That Worked Normally at Installation Later Perform Malicious Actions?

When a tool is first connected, the AI Agent retrieves data as requested, and the team responsible for deployment verifies the results. But if the provider later changes the tool description, will anyone at the company know? And who should decide whether it can remain in use? An AI Agent selects tools based on the provider’s written descriptions, so when enterprises connect tools through protocols such as MCP (Model Context Protocol), they cannot assume those tools will remain safe simply because everything worked normally at installation.

Intellicon Solutions believes security checks should not end at acceptance testing; ongoing oversight is also needed in everyday use. Someone needs to review tool changes, outbound connections must be limited to approved purposes, and long-term memory needs an owner. Even if the model itself has undergone safety training, enterprises should still verify which data a tool can read, where it can send that data, and which actions require human approval first.

Are the Original Checks Still Enough If a Tool Changes After Installation?

According to iThome, a draft attack taxonomy from the U.S. National Institute of Standards and Technology (NIST) warns that attackers may initially provide a legitimate MCP server, then modify its tool descriptions after users install it, inducing an AI Agent to perform malicious actions without obtaining the user’s consent again. The draft also describes tool poisoning: hiding malicious instructions in tool descriptions to influence which tools an AI Agent selects and how it plans its execution steps. Research cited in the draft indicates that even safety-trained models may not be able to resist attempts to induce malicious actions through legitimate tool interfaces.

The draft recommends using cryptographic signatures and immutable version identifiers to verify the source and version of tool definitions, along with sandboxes to run tools in isolated environments. However, the document is still open for public comment, and the report provides neither attack success rates nor cases of affected enterprises. It would therefore be wrong to conclude that every tool is compromised. Enterprises can use these recommendations to compile a list of questions providers should address during acceptance testing.

We recommend that IT maintain a tool inventory that records each provider, business purpose, accessible data, required external domains, and the person responsible for reviewing changes. Next, ask providers for evidence of how they verify tool definitions and isolate tool execution environments. Also clarify how they notify customers of updates and compare changes between versions. “The integration is working” is not a sufficient answer.

If a tool’s description, data access scope, or outbound connections change, we recommend having IT review the changes first, followed by the business owner confirming their intended use. Affected work should be paused until this review is complete. Before resuming use, test the tool against the original business scenarios to confirm that it neither accesses data beyond its permissions nor introduces unapproved actions.

Why Manage Outbound Connections If a Tool Can Only Read Data?

Suppose a procurement department wants an AI Agent to look up historical purchase prices and prepare recommendations based on price comparisons. Although this task requires data access, that does not mean the AI Agent should be allowed to modify procurement records or send the entire price list to an external service. In this hypothetical scenario, we recommend that the business manager first identify the required data and its permitted uses when designing the workflow, then have IT configure access accordingly.

In EgentWrX, for example, the egress allowlist blocks all outbound connections by default, with administrators allowing domains individually. The “Company Internal Database” integration is read-only and cannot write data. Enterprises can use these mechanisms to limit the scope of operations, but they should not assume that every MCP tool is read-only like the database integration. Nor should outbound connection restrictions replace tool definition verification or sandbox isolation.

Read-only access can prevent data modification, but it cannot guarantee that data will not leave the organization. We therefore recommend that IT also verify which allowlisted services may receive query results, while business managers approve which fields may be shared with which recipients. If a workflow needs to send data outside the company, arrange for someone to verify the recipient and content before transmission. A domain’s presence on the allowlist should not mean that every query result may be sent to it.

As Memory Grows, How Should You Handle Abnormal Behavior?

A single attack can also affect later tasks. The draft discussed in the report warns that attackers may leave malicious content in long-term memory. As task-related data accumulates, an AI Agent may also become less consistent in following earlier instructions and constraints. We therefore recommend including memory in routine reviews, particularly records that could influence decisions in future work.

For example, if a procurement employee receives an exception to handle an urgent request, the enterprise should check whether the AI Agent might store that one-time exception as a rule that applies to future tasks. We recommend that business managers and IT jointly establish rules for writing to memory. For content involving company policies, sensitive background information, or anything that could affect future decisions, these rules should specify who must verify it and on what basis before deciding whether to retain it.

EgentWrX lets users view, edit, and delete memories, and supports tiered policies that allow direct writes, require review, or require approval. Enterprises can choose how to manage memory based on the risk of its contents. However, these management features do not automatically detect memory poisoning, in which malicious content is written to memory to mislead future work. The person responsible must still review the content and its source.

If an AI Agent repeatedly behaves abnormally across different tasks, we recommend pausing the affected work, recording the circumstances in which the issue occurred, and preserving suspicious content. The person responsible should then review, correct, or delete the relevant memories and retest using the same scenario. The draft also warns that if the model itself, or components used to subsequently adapt it, have been poisoned, restarting or restoring the system may not remove the problem. If abnormal behavior persists, IT and the provider should investigate. Deleting memories alone is not enough to conclude that the risk has been eliminated.

FAQ

Is passing tests at installation enough for an MCP tool to pass a security acceptance review?

No. An attacker may modify the tool description after installation, so the original tests cannot establish that later versions are also safe. We recommend assigning someone to review changes, requiring the provider to supply version and update records, and repeating acceptance testing with real business scenarios.

Does making a database read-only prevent data leakage?

No. Read-only access restricts data modification; it does not prevent retrieved data from being sent outside the organization. In addition to confirming that database writes are prohibited, check which fields the tool can read, which services may receive query results, and who verifies the content before it is sent.

Can we resume use after deleting memories or restarting the system when abnormal behavior is detected?

No. Neither deleting memories nor restarting the system is sufficient to establish that the risk has been eliminated. We recommend pausing affected work, documenting the circumstances of the abnormal behavior, addressing suspicious memories, and then retesting. If the problem persists, have IT and the provider investigate other possible causes.

References

30 minutes to map out which work to hand to AI first

Want every employee to have their own AI teammate?

A consultant will be in touch shortly.

ISO/IEC 27001:2022 Certified ISMS