A few weeks ago I was in the middle of a long conversation with Claude — working through a complex client memo — and it just stopped. Ran out of context. Then I looked at the API usage and realized I’d burned through tokens faster than I expected. That moment made me realize: I was using AI every day and I didn’t actually understand the unit I was paying for.
That unit is a token. And once you understand what it is, a lot of things about AI costs, vendor contracts, and tool behavior start to click into place.
So What Actually Is a Token?
A token is not quite a word and not quite a character. It’s something in between. In English, one token is roughly three-quarters of a word. So a 100-word paragraph is about 130 tokens. Punctuation counts. Spaces count. The percentage sign in “100%” is its own token.
Before any AI model processes your prompt, it runs it through a tokenizer — software that breaks your text into these chunks, assigns each one a number, and hands that stream of numbers to the model. The model never actually reads words. It reads numbers. When it responds, it predicts the next number in the sequence, one token at a time, until it’s done. Then everything gets translated back into text you can read.
That’s the whole process. Your words become tokens. Tokens become numbers. Numbers become a response. Response becomes tokens again. And you get billed for the tokens on both ends.
Input vs. Output: The Asymmetry That Matters
Here’s the part most people miss. Input tokens — the prompt you send — cost significantly less than output tokens — the response you receive. Typically three to five times less. So if you paste in a 10-K filing and ask for a summary, you’re paying less for the filing than you are for the summary itself.
That asymmetry changes how you should think about AI interactions. Loading a model up with rich context is relatively cheap. Getting long, detailed responses back is where the meter really runs.
Context Windows: How Much the Model Can See at Once
Every model has a context window — the maximum number of tokens it can hold in its working memory at once. Early ChatGPT had a 4,000-token limit. That’s about three pages of text. Today’s frontier models have context windows of 200,000 to one million tokens. You can literally feed an entire novel into some of them.
But bigger windows come with a catch most users don’t realize: in a chat interface, every message in your conversation gets resent as input tokens every single time you send a new message. A conversation that’s been running for an hour isn’t just the latest exchange — it’s the entire history, resubmitted with each turn. Long threads get expensive fast.
The Price War — And What It Means for You
Token pricing has dropped dramatically. Claude Opus went from $15 per million input tokens to $5 — a 67% cut — in a single model generation. That’s driven by competition, better hardware, and pressure from cheaper alternatives like xAI’s Grok and open-source models out of China.
Every major provider — Anthropic, OpenAI, Google, xAI — prices on a per-million-token basis, split between input and output. The flagship models are expensive. The budget models are remarkably cheap. And firms that route the right tasks to the right model tier are finding they can cut AI spend by 80% or more without meaningful quality loss for routine work.
One More Thing: Thinking Tokens
Advanced reasoning models now do something called extended thinking — they work through a problem internally before responding. Those internal steps are billed as output tokens even though you never see them. Ask a flagship model a simple question and it may still generate thousands of thinking tokens behind the scenes. It’s worth knowing this before you route complex tasks to the most expensive model by default.
What Accounting and Finance Teams Should Do
- Track token usage by user and team. Most APIs return token counts in every response. If you’re building or overseeing an AI program, build that tracking in from day one. It tells you who your power users are and where costs are concentrating.
- Right-size the model to the task. Drafting a standard engagement letter doesn’t need the flagship model. Routing routine work to budget tiers is one of the highest-leverage cost moves available.
- Understand vendor token economics before signing. When evaluating AI tools, ask how they account for tokens. Seat-based pricing, credit systems, and token pass-through models all have different cost profiles depending on how heavily your team uses the tool.
- Use prompt caching. If your AI tool sends the same system prompt repeatedly — instructions, firm context, templates — caching means you’re not paying full price to re-send it every time.
- Start new chats for new topics. Sounds simple. Saves real money.
Key Takeaways
- A token is roughly three-quarters of a word. It’s the unit AI models use to read input and generate output.
- Output tokens cost three to five times more than input tokens. Long responses are more expensive than long prompts.
- Context windows have expanded to one million tokens — but long chat threads burn through them fast.
- Thinking tokens are billed even when you can’t see them. Flagship reasoning models think a lot.
- Token economics belong in AI procurement, budgeting, and governance conversations at every firm.
Want the CPE credit? Take the full lesson on EverydayCPE and earn 0.2 CPE credits: [lesson link]
Leave a Reply