The fact that OpenAI documents theirs is already a big improvement over Anthropic. But, also, the OpenAI tokenizer got more efficient when they last updated it, rather than less. https://mdstudio.app/o200k-base-tokenizer
Yeah, Anthropic's current tokenizer in Sonnet 5/Opus 4.8/Fable 5 is much worse than OpenAI's. Also, OpenAI has been using their current o200k_base from the day GPT-4o came out over two years ago. Just a few of my own tests:
- A ~2000-2002 legacy C++ game codebase at about ~90kloc: GPT 1.12M, Claude 2.2M
- A ~30kloc TypeScript codebase: GPT 260K, Claude 437K
In the end, GPT's current tokenizer is ~1.6x-2x better than Claude's current one, depending on your data. And you can check for free for both, for OpenAI just use the open-source libraries, for Anthropic - you have to use their count_tokens endpoint as they don't publish the tokenizer, but the endpoint is free (and allows requests over 1M tokens as well).
Interesting... Naively I'd assume you'd have a pretty unfair advantage on quality if you have materially more information dense tokens.
That doesn't really appear to be the case as GPT and Anthropic models appear evenly matched despite Anthropic encoding the same text into almost ~2x the tokens...
I'd also - naively - assume this would make training their models more expensive. Though inference now dominates, and they'd probably rather have more tokens than less (to charge you for them at future 80% margins).
more dense tokens means more stuff to fit into the embedding space which is per token, so more work to disentangle later
you use the wrong word
the Anthropic tokenizer is not worse, its more expensive/verbose
So, worse? Because we benchmark off token use when talking about token use, and everyone else understood that.
It’s better for them
We've switched the default model in playcode.io among Opus 4.8, Opus 4.6, Sonnet 4.6, and Sonnet 5. I must admit, Opus 4.8 is quite expensive, and the costs accumulate quickly. Opus 4.6 is about 50% cheaper, while Sonnet 5 is significantly more affordable. According to the data, Sonnet 5 is about 2-3 times cheaper. Fable 5 is unaffordable at all...
Today, I tested Sol 5.6 on various tasks. It performs similarly to Opus 4.8 but is still noticeably more expensive than Sonnet 5. Although Sonnet 5 isn't the top model, it's quite effective for creating typical websites for small and medium businesses. However, they will increase the price starting September 1, as their free offer is ending.
I'm also actively testing Grok 4.5. There's something promising about it. The design is mediocre, in my opinion, but it operates quickly and reliably without any deadloops. Usually, Grok models would fail or loop, but this one is stable.
Overall, I really want a benchmark based on real tasks.
The real elephant in the room is pricing for KV cache writes and reads. That makes all the difference for tasks with large context.
[dead]
Yes, and verbosity (thinking) is a huge factor.
I have reduced usage of Fable and Sonnet 5 to a minimum. Fable in particular is amazing at creative tasks, but not worth the cost for almost everything else. I can have Opus 4.6/4.7 running non-stop without hitting quota, vs maybe 20 minutes of Fable usage.
Well, I both agree and disagree with you.
On one hand, the price is just astronomical for Fable, well, not exactly astronomical, but I would say unaffordable. That is to say, so expensive that it is impossible to use.
But on the other hand, Fable is simply incomparable to anything else. I mean, it is just amazing. There is nothing even close to being equal to it.
Fable can solve the problems Opus couldn't. BUT most of the time I'm not having those kinds of problems.
I wouldn't say I'm doing anything groundbreaking but definitely at times obscure and that's when Fable has been able to dig me out of the rut. (the alternative I was actually following was reading textbooks myself to understand the domain better)
The real price is how often a model uses subagents that scan your whole repository multiplied by thinking level.
Honestly, I find performant pricing where they test each model on the same task is much more useful than figuring out the tokens or using the input token tax.
The reason being is that the only tokens I feel I really control are the input tokens, but the whole program seems to just run itself and they just charge you what they want to charge you and it’s more of a black box.
Very interesting article though.
My take from this is that Anthropic is screwing us again. I hope AMD shows them up again.
this is surprisingly high delta. to make matters worse, reasoning tokens account for the majority of tokens and they are completely opaque so it's hard to tell how much of that is prose or code
Are there any advantages of the new tokenizer? Does it have a larger or smaller vocabulary or just differently weighted?
Well, in my view, it's just the most ordinary manipulation to avoid creating unrest. There is most likely no improvement inside.
Of course, these are my guesses, but did anyone feel the difference in the transition from Opus 4.5 to 4.6? In my opinion, no. And it's unlikely to be a matter of the tokenizer.
This piece focuses on the cost differences from the tokenizer, which do matter, but I wish they emphasized more that even adding the tokenizer to your calculation doesn't provide you with a good way to calculate cost for agentic coding tasks.
Other traits where models differ that have an even greater impact on your total spend:
* How much context do they load in to solve a given task?
* How long do they spend thinking to get equivalent results?
* How many times do they stop and ask you for input, and are you there to respond to them before the cache runs out?
* Etc.
Incorporating the tokenizer just makes a very imprecise measurement of cost a little bit more precise, but in my own experience I have not found that the token cost is a significant driver of task cost whether or not you incorporate the tokenizer. Everything else about the model's behavior has a much larger impact.
An individual token, and the level of energy it represents (electricity, or relative effectiveness per model) increasingly seems the space of obsfucation.
This space can be increasingly avoided by becoming, and remaining, efficient and effective with prompts.
Tokenzier aside, a report shared on reddit found that the GPT 5.6 (edit: 5.5) series are incredibly thrifty with CoTs, resulting in cheaper bills than GLM 5.2 (let alone Opus/Fable): https://www.reddit.com/r/ZaiGLM/s/rUoG5adkPh
Chattiness remains an open issue for some of the SoTA open weights & (to a lesser extent) Claude.
That link does not mention 5.6.
Interesting. New models are estimated at ~5T params, so 45,000x increase over BERT base (110m). But vocab size of 200k, so only an increase of 7x over BERT base (30k).
[dead]