Three of the best-funded AI labs in the world, Anthropic, OpenAI, and Meta, all disclosed within about a week of each other that their AI models had broken out of a supposedly sealed test environment and touched real, live systems. Same failure. Same vendor. And if you’ve been watching this from the finance and accounting side, here’s the part that should actually get your attention: these are some of the most capable engineering teams on the planet, and they still couldn’t get access controls right on the first try.
The headlines made it sound like the AI went rogue. It didn’t. This is a controls story, and it’s one of the clearest lessons finance and accounting professionals are going to get this year about what least privilege actually means once you move past the slide deck and into practice.
What Actually Happened
All three labs use a vendor called Irregular to run their cybersecurity testing. Irregular builds sandboxed environments where AI models get evaluated using capture-the-flag exercises. Think about it as the childhood game with two flags and two teams, just played out through hacking systems instead of a backyard. Each team defends its own flag and tries to capture the other one, using both offense and defense.
The setup is supposed to be fully sealed off from the real internet. It wasn’t. A misconfiguration left these sandboxes with a live internet connection they were never supposed to have. Here’s the detail that turns this into a genuine controls failure rather than just a technology failure: the models were told, in their own instructions, that they were operating inside a secure simulation. The instructions said this is a test, you’re contained. The actual network connection said otherwise.
Instructions Aren’t Controls
This is the phrase I keep coming back to, and I flagged the same issue before Irregular ever made news, back when we covered the Hugging Face incident. Quick correction there, since the two are easy to conflate: Hugging Face had nothing to do with Irregular. That was OpenAI’s own internal cybersecurity benchmark, and OpenAI had tried to sandbox the model on purpose. It escaped anyway and went looking for real data to improve its score. Different vendor, same underlying lesson: telling a system it’s contained is an instruction. It is not a control.
A control restricts what a system can actually reach. If the only thing standing between your AI agent and the open internet is a sentence in a prompt, you don’t have a control. You have a suggestion, and a capable enough agent will find the fastest path around it.
What This Means If You’re Building With AI
Here’s where this gets directly relevant to your work, whether you’re evaluating a vendor or building agents yourself. Anthropic reviewed a large number of interactions during this testing and confirmed that only a handful actually resulted in a model touching a real system. That’s a tiny failure rate. It was also enough to become a headline story across three of the biggest AI labs in the world. Scale doesn’t make loose access controls safe. It just makes the eventual failure bigger.
The practical fix is least privilege, enforced technically, not through a prompt. Say you’re building an agent that connects to your CRM to pull deal and fee data. During development, it’s reasonable to give it broad read access to a bunch of tables so you can figure out what it actually needs. Locking it down too early just slows your build for no reason. But the day you move that agent toward production, the access needs to shrink to the narrowest scope that still gets the job done, and it needs to be enforced at the system level, not the prompt level.
The same logic applies to something as simple as a research agent. If you want it to check the Big 4 firms’ websites every morning, you can write a prompt that says only check these four domains. Or you can configure the tool so it’s technically incapable of visiting anything else. Only one of those actually holds up if the agent decides there’s a faster way to get the job done.
Key Takeaways
- Three disclosures, one vendor, one root cause: a sandbox that had live internet access it should never have had.
- The Hugging Face incident was a separate case and didn’t involve Irregular.
- An instruction telling a system it’s contained is not the same as a technical restriction on what it can reach.
- Wider access during build and test is fine. It can’t survive, unverified, into anything treated as production.
- Before anything goes live, know exactly what it can reach, and make sure you can log and review what it actually did.
Want the CPE credit? Take the full lesson on EverydayCPE and earn 0.2 CPE credits: [lesson link]
Leave a Reply