One of my clients runs the same 50 questions through their AI system every day. They use LLM judges to score the answers, and they watch for drift. As models change, they can see whether it’s affecting what their clients get back.
That sounds like overkill until you look at the calendar. On January 14, 2027, in the middle of year-end close, OpenAI turns off GPT-4o and GPT-4.1 in its API. If your team built anything on those models in 2024 or 2025, a reconciliation helper, an invoice-coding automation, a chatbot that answers vendor questions, it stops working that day.
The harder question isn’t whether it still runs after you switch. It’s whether the answers are still right, and how you would know.
What’s actually going away
On October 14, OpenAI announced it will remove GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini from its API on January 14, 2027. The recommended replacement is GPT-6 Luna.
That’s one of four dates in under three months. A batch of older models comes off on October 23. The original GPT-5 models go on December 11. And from January 6, you can’t start new fine-tuning jobs at all.
If your team only uses ChatGPT in the browser or the app, you won’t notice. OpenAI already pulled these models from ChatGPT in February, and at that point said only 0.1% of ChatGPT users were still choosing GPT-4o on a given day. People moved on. Integrations often don’t, because nobody revisits an automation that’s working.
Why this is a finance problem, not just an IT problem
Once an AI tool produces numbers, classifications or summaries your team relies on, it’s doing the job of an automated control. With most automated controls, the logic stays fixed until your team decides to change it. With AI, the engine underneath can be swapped on someone else’s schedule.
You don’t need a new framework. Before an ERP upgrade goes live, someone tests that the three-way match still blocks the mismatched invoice. A model swap is an ERP upgrade the vendor schedules for you.
Three ways the model changes
Forced. The vendor retires the model. You get a deadline and no vote.
Chosen. A newer model is cheaper, faster or better, and someone wants to move. Here you get a vote, so you need a basis for it.
Silent. This is the one most teams miss. A lot of tools call a short name like “gpt-4o.” That name is a pointer, and the vendor decides which dated version it points to. The vendor can move it without a press release, and your outputs can change without anyone on your side making a decision.
The fix is pinning: point your solution at a specific dated version, so it only changes when you decide. The tradeoff: when that pinned version is retired, you need a maintenance process ready.
What a replacement changes
A new model isn’t a drop-in copy. GPT-6 Luna is a reasoning model that works through a problem before answering, at medium effort by default. That can change the speed, the cost and the shape of the answer: longer responses, different formatting. If another automation reads that output, it can break even when the answer itself is right.
Cost is the hidden one. A newer model can have a lower list price and still cost more per task, because it does more thinking or uses a different tokenizer. Track cost per task, not price per token.
Four pieces to set up
1. An inventory with an owner. List every place your team uses AI through an integration. Record the exact model, the owner and how much the business depends on it. Put retirement dates on the same calendar as license renewals. For vendors with AI built in, ask: which model does this run on, when will you switch it, and how are you testing for model drift?
2. A test set. Pull 20 to 50 real cases from the last few months where you know the right answer. Include the awkward ones: the partial payment, the vendor with two names, the invoice with a handwritten note. Decide what passing looks like before you run the test, not after. Then test the whole chain, not just the AI step.
3. Log the model version on every output. If a number in your close came from AI, you should be able to say which model produced it and when. Without that, you can’t tie a change in results to a change in the model.
4. Ongoing monitoring. Keep running the test set after the switch. Hand-check a small sample every month, set thresholds on override and error rates, and flag automatically when they move.
Key takeaways
- The model under your AI solution will change, whether you choose it, the vendor forces it or it happens silently. Pin a specific version.
- Treat a model change like a system upgrade on an automated control. Test with your own cases, not the vendor’s benchmarks.
- Build a test set of 20 to 50 real cases and set the pass bar before you run it.
- Log the model version on every output and watch override rates.
- Give every integration an owner and put every retirement date on the calendar.
If anything on your team runs on GPT-4o or GPT-4.1, the deadline is January 14, 2027. Build your test set this month.
Want the CPE credit? Take the full lesson on EverydayCPE and earn 0.2 CPE credits: Your AI Has a Shutdown Date
Leave a Reply