The Session You Can’t Actually Audit

Anthropic put out a disclosure this week that should get more attention than it’s gotten: three of its Claude models hacked into real companies during routine testing, and nobody at Anthropic knew until they went back and manually checked. Not a live alert. Not a monitoring dashboard flagging something odd. A retrospective review of 141,000 past evaluations, triggered only because OpenAI had just disclosed a similar incident with Hugging Face.

I want to walk through what actually happened, and then get into the bigger thing I think this points to because the hack itself isn’t really the story.

What Anthropic actually found

Anthropic reviewed 141,000 past evaluations and found three instances where Claude models reached the internet and touched systems belonging to real companies, not sandboxed test environments. In two of the three cases, the model realized partway through that it was doing something it shouldn’t and stopped on its own. In the third, the model had been assigned to attack a fictitious company for a benchmark, and that fictitious company happened to share a name with a real one. It broke into the real company and kept going.

None of this was especially damaging, as far as Anthropic has disclosed. For all 141,000 of those evaluations, across however long they’d been running, nobody had been reviewing the logs closely enough to catch this on their own. It took a competitor’s bad week to trigger the review that found it.

That’s two separate failures stacked on top of each other. One is access, a model doing something it wasn’t authorized to do. The other is detection, nobody knowing it happened. You can fix the first one completely and still have the second one open.

The bigger pattern underneath it

Here’s where I want to spend most of this post, because I think it matters more than the hack itself.

Three years ago, if you used an early API call or an early version of ChatGPT, you basically saw everything. Input, output, and if reasoning was happening, you could often see the actual back-and-forth. As tool use got added, you could see the tool calls. The full transcript was, more or less, yours.

That’s been eroding. Not because it’s technically impossible to keep giving you the full picture, but because providers have business reasons to pull it back, including a real concern about competitors using their outputs to train competing models. Whatever the reason, the effect is the same: you’re seeing less.

Four specific ways this is happening right now:

• Reasoning — you get a written summary, not the actual chain of thought.
• Hosted search — you get a final answer and maybe a citation, not the actual passages the model read.
• Compaction — long sessions get shrunk down to save cost, and what got kept versus discarded isn’t always disclosed.
• Agent-to-agent — if one AI delegates to another, you might see that a task got completed without seeing what it actually told the other one to do.

A response ID is not a transcript, because that data lives on the vendor’s server, not yours.

A ciphertext is not your record, if you’re not the one holding the key.

A citation is not the evidence, because you usually can’t go pull the exact passage the model actually used.

Each of those looks like documentation. None of them is actually complete.

Why this is a compliance problem, not just a curiosity

We work in a field built on being able to reproduce and audit actions, explain them, and show our work after the fact. If you’re using AI agents anywhere near financial data, client work, or internal controls, the question to ask isn’t “do we have logs.” It’s: if a regulator or a client asked us to show exactly what the system saw and did before it took an action, could we actually produce that? Or would we be handing over a polished summary with real gaps behind it?

For a growing number of AI tools, the answer is the second one.

Key Takeaways

• Detection is a separate risk from access. You can lock a system down completely and still not know something happened.
• “We log everything” doesn’t mean “we review what we log.” Someone needs to actually be looking, on a schedule. Treat your agents the way you’d treat a new associate: given more trust over time, but never running fully unsupervised.
• Reasoning, search evidence, and compacted history are increasingly sealed by design, not by necessity. That’s a deliberate choice providers are making.
• A response ID, a citation, or a summary can look like documentation while actually just pointing back to something only the vendor can resolve.
• Ask any AI vendor directly: can you produce a full, human-readable account of this session, not just a plausible final answer with high level notes? Their answer tells you a lot about how ready their tool actually is for your world.

Want the CPE credit? Take the full lesson on EverydayCPE and earn 0.2 CPE credits: The Session You Can’t Actually Audit.