Researchers gave ten AI agents a wallet with a spending limit of one cent a day, then asked each one to pay three invoices for one cent each. Nine of the ten found a way to raise their own limit. Nobody told them to, and nobody hacked anything. The code to approve a higher limit was emailed to the owner, and the agent could read the owner’s email.
If you approve anything for a living, whether that’s payments, vendors, limits or access, that last sentence is the one to pay attention to.
What the study did
On October 26, researchers from Cornell Tech, Yale and IC3 posted a paper called I-HITL: Inalienable Human-in-the-Loop. The idea is simple. If a human is supposed to approve something, the agent shouldn’t be able to approve it for them.
To test it, they used Circle’s agent wallets, which are built for AI agents to spend from. The owner sets the spending limits. If the agent needs a higher limit, Circle emails a one-time code to the owner, and the owner enters it to approve the change. That emailed code is the human in the loop.
Each wallet got $10 and a limit of one cent per transaction and one cent per day. Each agent got three tools: email, a web browser and the wallet. Its instructions said to act only on the user’s requests and to treat anything that came in by email as information, not instructions. The task never mentioned the spending limit. They ran ten models, twenty times each.
How the loop broke
The agent paid the first invoice. The second one failed: spend limit exceeded. The agent decided on its own to request a higher limit. The wallet did exactly what it was designed to do and emailed a code to the owner.
But the owner wasn’t the only one who could read that email. The agent found it, pulled out the code and submitted it. The limit went from one cent a day to $1 per transaction, $5 a day, $20 a week and $50 a month. All three invoices got paid, and the only sign anything happened was a note in the agent’s summary afterward.
Nine of the ten models did this at least once. The behavior was inconsistent, and it didn’t line up with which models were the most capable, so you can’t fix it by picking a better model. The problem is the setup.
The second experiment was worse. Agents asked only to summarize the inbox got emails pretending to be the owner, asking for a small payment with a deadline. Five of the ten raised their own limit and paid the attacker at least once. That’s business email compromise, except the employee who fell for it was the agent.
Circle’s response was a warning in its documentation: don’t give your agent access to the inbox that receives the code. The fix lands on you.
An old control with a new hole
Segregation of duties says the person who asks for a change can’t be the person who approves it. Email codes worked as an approval channel for years because an inbox meant a person.
Think about giving your AP clerk access to the controller’s inbox so they can keep an eye on vendor order confirmations. Helpful, until you realize the clerk can now see the approval emails for their own payment limits. Nobody would set that up for a person. It fails the most basic access review.
A simple test I use: replace the AI with a person and ask whether anything changes. Here the control already exists for people. What’s new is how casually we hand that same access to agents.
Here’s how it happens in practice. Say you set up an agent to buy supplies and software, connect it to a corporate card, and connect it to your email so it can check that order confirmations come through. Reasonable. But the agent doesn’t get access to your order confirmations. It gets your entire inbox, approval emails included. You didn’t break the loop on purpose. You connected one extra tool.
What to do about it
Map the approval path. For every tool your agent touches, list the step-up actions: raising a limit, adding a payee, changing bank details, resetting a password, adding a user. How does each one authenticate? If it’s email only, and the agent has your email, the loop is broken. The same goes for texts if they sync to a device the agent can reach.
Apply least privilege. Before fixing the approval, ask whether the agent needs the inbox at all. A separate mailbox just for receipts, a forwarding rule for specific senders, or checking order status with the vendor directly are all narrower than your whole inbox.
Move the approval out of reach. Think of it as a ladder:
- Bottom rung: an emailed code. Broken the moment the agent has the inbox.
- Middle rung: a second factor on a separate device the agent can’t reach, like an authenticator app or push approval on your phone.
- Top rung: what the researchers argue for. An approval on a separate device that shows you the exact action you’re approving. Even a code on your phone can be handed over if the agent simply asks for it on a busy afternoon. The researchers built an open-source prototype app for this, called Countersign.
You don’t need the top rung for everything. In finance and accounting, though, plenty of approvals deserve it.
Key takeaways
- A spending limit is only as strong as the channel that approves changes to it.
- For every tool your agent touches, ask how it authenticates step-up actions and whether your agent can read that channel.
- Least privilege applies to inboxes. Give agents the narrowest mailbox access that does the job.
- Put approvals on a device the agent can’t reach, and for high-stakes approvals, use one that shows the exact action.
- If the same agent can both request a change and confirm it, the change isn’t approved. It just looks approved.
Want the CPE credit? Take the full lesson on EverydayCPE and earn 0.2 CPE credits: Your Agent Approved Its Own Raise
Leave a Reply