Yesterday I was talking with someone setting up their first AI chief-of-staff agent, the kind that starts your day by pulling together your calendar, your meeting notes, and your open threads. When they built it, they gave it access to every connector on their computer. Based on a benchmark Amazon just published, that’s the wrong way to set one up.
Amazon Science ran eleven frontier AI models through more than 2,000 real business tasks: standard operating procedures written by actual experts, across twelve domains including finance, healthcare intake, content moderation, and warehouse inspection. In their words, the main finding was that capability is not free.
Here’s what that means. In one test, an agent needed six tools to complete a task. Amazon gave it those same six tools, but buried them among twenty extra tools it didn’t need. The agent’s success rate dropped by roughly half. It didn’t just ignore the extra options. It explored them, burned tokens on things that weren’t relevant, and came back with results that missed the point.
Apply that to the chief-of-staff agent. If it only has access to your calendar and your meeting notes, it can do a clean morning summary: what’s on your schedule, what came up in recent meetings, who you’re talking to today. Give it your CRM too, and it might pull in HubSpot activity that has nothing to do with you, because someone else on your team touched a deal overnight. The extra access doesn’t make the agent smarter. It gives the agent more ways to go looking in the wrong place.
Newer models don’t automatically win
The benchmark also found that no single model performed best across every task. On some procedures, Claude Sonnet outperformed Claude Opus. We’ve seen the same pattern in other benchmarks that test models on finance-specific work. If you’re deciding which model your firm pays for, the takeaway is that newest and biggest isn’t a safe default. You have to test it against your own tasks.
Across the twelve domains, success rates ranged from 57% to 100%, and the model-and-agent combination that performed best on one procedure often performed worse on the next. Picking a good model for one task doesn’t mean that choice will hold up on a different one.
Development and production aren’t the same test
I’ve run into a version of this with my own setup. I built a chief-of-staff agent for myself that runs on Obsidian and keeps my notes in markdown. I’ve tested it against both local models and frontier models. It works for me day to day, but when someone asked if they could use it too, I told them it wasn’t ready. There are small things that need tweaking and edge cases that don’t resolve cleanly. That’s manageable when I’m the only user and I know where the problems are. Handed to someone else at scale, those same issues turn into real complaints. That gap, between something that works for you and something that’s ready for other people to use, is the real work of getting from development to production.
Treat tool access like least-privilege access
Amazon’s fix is the same principle we’ve applied to data access all week: least privilege. A new hire gets access to the systems their role actually needs, not the whole company. Agents should be built the same way, scoped to one task, with access only to the tools that task requires.
Before approving an agent for anything that touches money or client data, ask three questions. What’s the complete list of tools it can access for this task? Can you see a success rate broken out by procedure, not a single blended average? And when the model underneath the agent changes, is it re-tested against a set of prompts you already trust, rather than assumed to be an improvement?
Key Takeaways
- Bigger or newer models don’t automatically perform better. Test them against your own tasks before you rely on them.
- Fewer tools produce more correct outcomes. Give an agent only what the task requires.
- Tool scoping is a control, not just an engineering decision, and it’s a big part of what gets an agent from a working prototype into something other people can actually use.
Want the CPE credit? Take the full lesson on EverydayCPE and earn 0.2 CPE credits: Capability Isn’t Free
Leave a Reply