Power up. From supercharging employee productivity and streamlining processes to inspiring innovation, Microsoft’s AI is designed to help you build the next big thing. No matter where you're starting, push what's possible and build your way with Azure's industry-leading AI. Check it out.
AI agents might be taking on more responsibilities at work, but maybe think twice before handing over full access to payments.
A new paper from researchers at Princeton University and the Sentient Foundation found that certain agents—AI systems that can act beyond the realm of a chatbot—could be vulnerable to memory attacks that trick them into handing over cryptocurrency.
Targeting agents created with the platform ElizaOS, the researchers were able to implant false memories or “malicious instructions” that manipulated shared context in a way that could lead to “unintended asset transfers and protocol violations which could be financially devastating.” They wrote that the vulnerabilities point to an “urgent need to develop AI agents that are both secure and fiduciarily responsible.”
The research comes as payments companies like Visa, Mastercard, and PayPal are beginning to roll out new tools for agentic payments. While these tools aren’t the same as the open-source system targeted in the study, Himanshu Tyagi, co-founder of Sentient, said the research raises questions about the type of security issues at play as companies push forward into autonomous transactions.
“Across the world, all the payment providers are moving in that direction where you can delegate your payments to these agents, and the question is, what vulnerabilities are we walking into and what can happen?” Tyagi told Tech Brew. “In general, an autonomous agent is a very risky thing. There are many new attack vectors that people have not thought about, which we are surfacing.”
Tyagi said the paper focused on ElizaOS because it’s “the most popular open-source agentic framework in crypto,” and on cryptocurrency because its traders have most readily embraced these types of autonomous agentic payments.
While these agents do protect against basic prompt injection attacks—inputs designed to exploit the LLM—more sophisticated actors might be able to manipulate the stored memory or contexts in which these agents operate. The researchers designed a benchmark to evaluate the defenses of blockchain-based agents against these types of attacks.
They also argued that the vulnerabilities extend beyond just cryptocurrency-based or even financial agents: “The application of AI agents has led to significant breakthroughs in diverse domains such as robotics, autonomous web agents, computer use agents, and personalized digital assistance. We posit that [memory injection] represents an insidious threat vector in such general agentic frameworks.”
ElizaOS creator Shaw Walters told Ars Technica that agent administrators can implement controls that require validation and authentication before agents make payments. But Walters said the research does raise questions about vulnerabilities as agents gain more control of computers on which they operate.
The paper’s lead co-author, Atharv Singh Patlan, also counter-responded to Walters in Ars Technica: “Our attack is able to counteract any role-based defenses. The memory injection is not that it would randomly call a transfer: It is that whenever a transfer is called, it would end up sending to the attacker’s address. Thus, when the ‘admin’ calls transfer, the money will be sent to the attacker.”
This study isn’t the first to point out how AI agents may be subject to financial scams. As more and more companies turn to this technology, security experts are finding many ways in which they’re vulnerable to attack, which can be particularly detrimental given agents’ autonomous nature.
For companies adopting these tools, Tyagi recommends choosing only vetted agent frameworks. “We are very early in the auditing era for these agentic frameworks,” Tyagi said. “If you don’t know better, go with more audited or more researched frameworks.”