Anthropic let AI agents trade on behalf of 201 of its own employees. The agents were decent negotiators, and most people still got the wrong thing. About 85% of the value that went missing was lost before a single trade happened, because the agents misread what their people wanted. The agent can be capable of the task and still be working toward the wrong goal.
What the experiment did
Anthropic’s economics team called it Project Swap. Each employee brought one book they wanted to give away. Claude had a short chat with each person about what they like to read and what they wanted that summer. The typical person typed about 200 words. From that chat, Claude ranked every book in that person’s office pool.
Separately, each person ranked ten books themselves. The agents never saw those rankings. That’s the answer key, and it’s what lets you grade the agents against what people actually wanted instead of what the agent thought they wanted.
Then the agents traded in a shared channel, proposing swaps, accepting or rejecting them, and sometimes setting up multi-party deals. Half were told to be ruthless and just get their person the best book. The other half were told to be prosocial and try to make everyone happy. Anthropic ran 205 versions of this, changing one thing at a time, like which model the agents used.
Where the value went
Put the scores on a 100-point scale, where your favorite book is 100 and your least favorite is zero.
- 89: the best possible result, using everyone’s real rankings.
- 60: a perfect assignment using Claude’s rankings, graded against what people actually wanted. No haggling, no bad trades.
- 55: what the agents actually got. That’s roughly your fifth choice out of ten.
The drop from 89 to 60 is the misunderstanding. The drop from 60 to 55 is the trading. Even perfect negotiating, working from Claude’s read on people, couldn’t get past 60.
Claude’s rankings matched people’s own on 61% of book pairs. A coin flip gets you 50%, and ranking by popularity got about 53%. Better than the alternatives, but a long way from knowing someone.
A bigger model didn’t fix it
Graded against Claude’s own rankings, bigger models looked a lot better. Haiku, the smallest, scored 75. Opus scored 88. When Opus agents traded against a smaller model, Opus always came out ahead.
Then Anthropic graded the same runs against what people actually wanted, and the gap between the smallest and biggest model shrank to about one point. The better model negotiated harder and won better deals, on the wrong target.
The instructions told the same story. Telling an agent to be ruthless added about 2 points on Claude’s rankings. Upgrading from Haiku to Opus added about 12.
What Anthropic recommends
Their recommendations are aimed at people building agents. Check that the agent understands the person it represents. Show the person a few sample decisions before the agent acts on its own, and let them correct it or opt out. Keep what the agent is doing visible.
What this means for finance teams
Think about an engagement letter. Most engagements that go sideways don’t go sideways in fieldwork. They go sideways in scope. You can do everything perfectly to the letter and the client is still unhappy, because what they wanted and what the letter said weren’t the same thing. That’s my framing, not Anthropic’s, but it’s what happened on these trading floors.
Two practical moves:
Make the agent show you its understanding before it acts. This one comes straight from Anthropic’s recommendations. Don’t just ask for a plan. Ask it to repeat back, in its own words, what it thinks you want, and fix what’s wrong. Then ask for a few sample calls, like its top three vendors and why, before it runs the whole job.
Give it more to work with. Say what good looks like in one sentence, rank your priorities and add the context it can’t know. Compare “get me three quotes for a laptop refresh” with “three quotes under $1,500 a unit, from our two approved vendors, three-year warranty, no refurbished, no touchscreens.” The first one leaves the agent guessing about everything that matters to you.
Detail helps, but it won’t get you all the way there. That’s why the read-back matters more.
Key takeaways
- In Anthropic’s agent market, about 85% of the lost value came from agents misunderstanding people. About 15% came from the trading.
- A better model negotiates harder for whatever it thinks you want. It can’t fix the wrong target.
- Before an agent acts, have it show you its understanding: a summary of what it thinks you want and a few sample decisions.
- Give it more detail, and still do the checks.
- Before you pay for a bigger model because the output is off, check whether the agent understood the task.
Want the CPE credit? Take the full lesson on EverydayCPE and earn 0.2 CPE credits: Your Agent Doesn’t Know What You Want
Leave a Reply