How AI Actually Fails — and Why “It Got It Wrong” Isn’t a Control

On structured tasks, even the best AI agents available today succeed a little under two-thirds of the time. That number moves constantly, and no model performs perfectly on any benchmark — in the real world, failure rates run higher than the benchmarks suggest. This isn’t a reason to avoid the tools. It’s a reason to get specific about how they actually fail, because “the AI got it wrong” isn’t specific enough to design a control around.

There’s a strange pattern in how people talk about AI: blind trust for some tasks, blind distrust for others, often for the same underlying tool. Neither instinct is useful. What’s useful is naming the actual failure modes, because once you understand the mechanics behind each one, you can build a real control instead of a vague sense of caution.

Hallucination is the one most people have heard of. It’s a byproduct of how these models get trained. During training, models get rewarded more for producing an answer than for declining to answer — that preference is baked in early, and reinforcement learning on top of it tends to make models more confident-sounding, not less. The model doesn’t know when it’s hallucinating. If it did, it wouldn’t. And because it doesn’t know, nothing about how it phrases a wrong answer signals that it’s wrong. It sounds exactly as confident as a correct answer. That’s what makes it dangerous — there’s no implicit trust signal in the way AI writes, even though the tone reads like there is.

Context window is the amount of text a model can actually hold and reason over in one conversation. Every time you send a new message, the whole conversation gets fed back into the model. That adds up fast, and once you exceed the model’s limit, the labs start managing it through things like compaction — quietly trimming or summarizing earlier parts of the conversation so it fits. You may never see this happen. You might scroll back in a long chat, notice a correction you made earlier, scroll back down, and watch the model make the same mistake again — because somewhere in between, a compaction happened and that detail got dropped. If you’re working through something material in a long session, that’s a real risk, and it’s often invisible.

Training cutoff is the date after which a model has zero built-in knowledge. This matters most when a model is deliberately kept offline — which, in finance and accounting, is often the right call, especially when you’re working with sensitive or PII data you don’t want touching the internet. But it means the model can be confidently wrong about anything after its cutoff: a new IRS rate, a new bill, a recent ruling. It won’t necessarily flag that gap. It’ll just answer.

The fix is matching your verification method to the actual failure mode. Worried about hallucination? Fact-check specific names, numbers, and citations — and watch for AI-generated sources showing up in research, since a lot of reputable outlets now block AI crawlers, which skews what gets surfaced. Worried about context window loss? Keep sessions shorter and restate anything critical rather than assuming it’s still in memory. Worried about training cutoff? Never assume the model knows something time-sensitive — pull the current rate or rule yourself and give it directly. This came up constantly during the “One Big Beautiful Bill” saga: ask an AI about it without feeding it the actual current text, and you were often getting a wrong answer stated with total confidence.

Key takeaways

• Hallucination is a confident, specific, wrong answer — not a flagged guess. The model doesn’t know it’s wrong until you check.
• Context window is the amount of information a model can hold in one conversation. Long sessions can silently drop earlier details through compaction.
• Training cutoff is the date after which a model has zero built-in knowledge, whether or not it says so.
• Design your review process around the real failure rate on structured tasks, not a general sense that AI is “probably right most of the time.”
• Match your verification method to the failure mode you’re actually worried about — and never assume a model knows anything time-sensitive without checking yourself.

Want the CPE credit? Take the full lesson on EverydayCPE and earn 0.2 CPE credits: How AI Actually Fails — and Why "It Got It Wrong" Isn't a Control