Why do LLMs need tokens?
At the start of July 2026, Anthropic released Sonnet 5. Not as good as Opus 4.8, but better than Sonnet 4.6. And obviously there are some new pricing tiers designed to subtly recoup some of the massive costs of developing and running a top-tier LLM.
I’ll leave a reference at the end about the benchmarks. Apparently the model is 3 to 10% better compared to Sonnet 4.6, which typically is probably unquantifiable to the normal user. No one is sitting in front of their terminal using Sonnet 5 saying “oh yeah, that response was better than Sonnet 4.6. In the range of…oh maybe 3 to 10%.”
Anyway, the pricing model is typical AI company bait and switch pricing that anyone who’s had an LLM vendor subscription for over a year is already well aware of. They use the now ubiquitous usage-based pricing model; Sonnet 5 has an introductory offer (aka “price to suck you in and get you hooked”) at USD$2 per million input tokens and USD$10 per million output tokens until 31st August 2026. The price then becomes USD$3 per million input tokens and USD$15 per million output tokens (a 30% increase). For context, Opus 4.8 is priced at USD$5 per million input tokens and USD$25 per million output tokens.
You might be confused about how token pricing is in the millions of tokens. And what even is a million tokens anyway? They’re good questions. I’ll answer them below and also try to shed some light on the reason why models are so token hungry.
The Memory Game
Using the earlier Sonnet 5 pricing model, we can create an example. A coding session to generate something takes 2000000 input tokens and gives 400000 output tokens.
(2,000,000/1,000,000 x $2) + (400,000/1,000,000 x $10) = $4 + $4 = a total price of $8
That example gives us the first things to talk about; what are input and output tokens and why are they treated differently? The basic customer-facing unit when using a model is a token. This is some text, broken up into chunks. I won’t go into too much detail, but each model has different ways of breaking text up into tokens (using a tokenizer). For example the quick brown fox jumps over the lazy dog can be broken up into tokens: The, quick, brown, fox, jumps, over, the, lazy, dog and .. The sentence becomes 10 tokens.
But why are they treated differently, like a phone call where you’re billed separately for talking and listening. The response is expensive because of how LLMs work and how they generate responses. The model can’t write a whole response in one go; it has to produce the response one token at a time in sequence. To generate a new token, the model needs to know if the new token fits with the previous collection of tokens in the response. There are ways that are used to make it more efficient, but that’s the general idea of how it works. When creating a response the model produces one token, feeds it back in and adds it to the response it’s constructing, then produces the next token. A way to think about it is that it’s like playing the “I Went to the Market” memory game. The example below shows this more visually.
| input | Summarise this in one sentence: The quick brown fox jumps over the lazy dog. |
| processing | The model formulates the answer |
| output | A fox jumps over a dog. |
The input is 16 tokens (punctuation is also a token), the output is 7. The processing part is where the model does some analysis on the input and checks the data it has to see if it can give an answer. How the output response is created:
There need to be 7 passes.
A- predicts that next token isfoxA fox- predicts that next token isjumpsA fox jumps- predicts that next token isoverA fox jumps over- predicts that next token isaA fox jumps over a- predicts that next token isdogA fox jumps over a dog- predicts that next token is.A fox jumps over a dog.
This means that a 7-token output isn’t “7 tokens’ worth of work”, it’s 7 separate full passes through the entire model. This means that each pass is slightly more expensive than the last because the sequence keeps growing. That compounding of the cost is the reason output pricing is almost 5x more than the input pricing.
There are ways to make models use fewer tokens. They’re all interesting with names like “speculative decoding” (smaller, faster models try to guess several tokens ahead then the bigger model checks/corrects them instead of generating the tokens itself) and “KV caching” (the model doesn’t compute the whole conversation for every new token, but reuses calculations from previous steps).
From a user’s point of view, there is a functionality called prompt caching that many AI assistants use which makes the process of constructing a response a lot less wasteful. For example a prompt that is used quite often can be cached and reused. But even with that, the token usage can become outrageous when you’re working with long documents and long answers. There are other controls a user can also use to minimise their token usage like setting token limits and sending signals to the model to stop generating tokens or batch processing where non-urgent work can be processed at a different time or with spare capacity. In fact, there are Reddit threads dedicated to just this, especially after a new token-based-pricing model is released.
Entering the Extraction of Value stage
LLM vendors like OpenAI and Anthropic have enormous R&D costs (costs to train or “teach” the models). To train a model reportedly costs well over USD$100M in compute. And then to run it (called inference) costs quite a lot too. The inference pricing has been dropping around 10x every 1 to 2 years. This is due to hardware and models becoming more efficient. Competition between vendors also pushes prices down. Those words sound sensible and nice together, but the reality is that a lot of the current LLM technology is wasteful.
For end users though, prices are still cheap for how much is being spent by LLM vendors. We have seen that models are expensive to train and run, but the vendors can afford this as they are getting heavily subsidised by billions of dollars of investment. But in the end, we’ll be strangled with limits in the “Extraction of Value” phase. With the enshittification character arc of modern software today, every new release of a model means that we get closer to the day when it’s no longer going to be feasible for a standard, everyday user to afford to use a decent model, with workable usage limits.