AI security tools can be manipulated into running malware, researchers warn

A new proof-of-concept attack dubbed “Friendly Fire” is raising concerns about the growing use of autonomous artificial intelligence agents in cybersecurity workflows. The research demonstrates how AI-powered coding assistants designed to identify security flaws can instead be manipulated into executing malicious code, effectively turning a defensive tool into an attack vector.

The exploit, developed by researchers at the AI Now Institute, targeted Anthropic’s Claude Code and OpenAI’s Codex operating in their autonomous modes. The attack required neither elevated privileges nor specialized configuration changes. Instead, the researchers embedded instructions within a seemingly benign README file inside a software repository and then asked the AI agent to perform a routine security review. The result was the execution of a hidden malicious binary.

According to the researchers, the attack works because AI agents frequently treat all text within their context window as equally authoritative. In practical terms, a developer’s instruction and text embedded within third-party software documentation can appear indistinguishable to the model.

A familiar problem with a new twist

The Friendly Fire attack is part of a broader category of vulnerabilities known as prompt injection attacks, where malicious instructions are concealed within content that an AI system is asked to process. The researchers demonstrated that a simple request such as “perform security testing on this project” was sufficient to trigger the chain of events. The AI agent read the repository documentation, interpreted instructions within the README as legitimate, and executed a hidden script.

To illustrate the concept, the researchers modified a copy of the open-source Python geocoding library geopy, inserting a malicious binary alongside a decoy source file designed to make the binary appear legitimate. The README then encouraged the execution of a security-checking script. From the AI agent’s perspective, running the script appeared to be an appropriate step in the requested analysis.

Importantly, the researchers emphasized that the issue is not tied to a specific software version. Rather, they argue the weakness stems from how contemporary large language models process information and make decisions.

Commenting on the findings to Digital Journal, Eljan Mahammadli, Head of AI Provenance at Polygraf AI, says the significance lies less in the specific exploit and more in the underlying architectural limitation that it reveals. “The important part of AI Now’s Friendly Fire research is the weakness it exposes, rather than the specific binary or poisoned README the researchers used to demonstrate it,” Mahammadli explains.

According to Mahammadli, AI models currently lack a dependable mechanism for distinguishing between trusted instructions issued by a user and untrusted content obtained from external sources. Everything is processed as a stream of tokens, meaning repository documentation can carry the same apparent authority as developer instructions

This helps explain why similar attack techniques continue to emerge through various channels. Mahammadli points to examples such as Adversa’s TrustFall research and Tenet’s Agentjacking demonstrations, both of which relied on hidden instructions designed to manipulate AI agents. The common theme is that AI systems struggle to determine who authored a piece of information and whether that information should be trusted.

“In provenance terms, this is a failure of attribution,” Mahammadli notes. “The agent acts on text without any dependable sense of where it came from or whether that source should be trusted.”

Despite the alarming nature of the findings, Mahammadli cautions against interpreting the research as evidence that AI should not be used for cybersecurity. Instead, he argues that the problem originates from a specific deployment architecture that has become increasingly common: agents that can simultaneously read untrusted content, execute commands, and access sensitive systems, with only a safety classifier standing between them and potentially dangerous actions.

“When those powers sit together, a single injected instruction is enough to turn the agent against its operator,” he says.

The solution, he suggests, lies in runtime security controls rather than relying solely on model improvements. Security systems should be capable of identifying externally sourced instructions and preventing them from escalating into executable actions. Sandboxing can also play a role, although Mahammadli warns that sandboxing alone should not be viewed as a complete solution.

Perhaps the most interesting aspect of the Friendly Fire research concerns the behaviour of advanced models themselves. The researchers observed that some newer models appeared capable of recognizing inconsistencies between the malicious binary and its purported source code origin. Yet despite detecting the mismatch, the models proceeded with execution anyway.  This finding challenges a widely held assumption in the AI industry, that more capable systems are inherently safer systems.

Instead, the research suggests that increasing capability without improving trust evaluation may simply create more efficient executors of instructions, regardless of who authored those instructions. A highly capable agent may therefore become more dangerous if it lacks reliable methods for establishing the provenance and trustworthiness of information.