How GPT's New Tokenizer Cut Costs in 40 Languages

Jonhisking · October 2, 2026

For years, anyone using GPT in Hindi, Tamil or Malayalam paid several times more than an English user for the same request. Then, with GPT-4o in May 2024, OpenAI switched to a new tokenizer, and for many languages the cost of a prompt dropped by more than half overnight. Here is how tokenizers work, what changed, and the measured effect on 40 languages.

How a tokenizer is built

GPT models use a method called byte-pair encoding (BPE). It starts from raw bytes and repeatedly merges the pairs that appear together most often in a large training text, until it has a fixed vocabulary of pieces. Frequent words end up as single tokens. Rare words are built from several smaller pieces, and in the worst case from individual bytes.

The key point: the vocabulary reflects the text it was trained on. If that text is mostly English, English words get their own tokens and everything else is pieced together from fragments.

Scripts outside the Latin alphabet suffer most. A character in Hindi's Devanagari script, or in Tamil or Malayalam, takes three bytes in UTF-8. If the tokenizer has few merges for that script, a single letter can cost several tokens.

cl100k vs o200k

OpenAI has used two main tokenizers for its recent models:

• cl100k_base, about 100,000 tokens, used by GPT-3.5 Turbo and GPT-4.
• o200k_base, about 200,000 tokens, introduced with GPT-4o and used by later GPT models.

Doubling the vocabulary left room for far more pieces from non-English languages, so whole words and common syllables in many scripts became single tokens.

Measured: the same prompt in 41 languages

We translated one 34-token English prompt (a request to summarize a customer email and suggest a reply) into 40 languages and counted tokens on both tokenizers. Each number is how many tokens the language needs compared with English.

What the numbers show

South Asian languages gained the most. Malayalam went from about 10 times the tokens of English to 1.85 times, an 82% cut. Kannada, Telugu, Gujarati and Tamil all fell by more than three-quarters. Hindi, spoken by hundreds of millions of people, went from 4.59× to 1.50×.

Middle Eastern scripts improved a lot. Arabic, Persian and Hebrew all roughly halved.

East Asian languages improved less, from a lower base. Korean went from 2.50× to 1.44×; Simplified Chinese is now almost at parity with English at 1.03×. Japanese, at 1.79×, is still one of the more expensive languages.

European languages changed least. They were already close to English on the old tokenizer. Some, such as Czech, Slovak and Polish, remain among the most expensive on the new one, at around 2×, because of their accented letters and long word forms.

The gap is smaller, not gone

On o200k, the same prompt still costs 15% more in Spanish, 44% more in Korean and twice as much in Czech as in English. Pricing is per token, so that difference goes straight to the bill, and it also means the context window fills up faster and usage limits are reached sooner.

Other providers are different

Anthropic, Google, Meta, Mistral and DeepSeek all train their own tokenizers, and their efficiency by language varies. A model with a lower price per token can still cost more for your language if its tokenizer splits your text into more pieces. When comparing providers, compare the cost of your actual text, not the price list. We compared GPT and Mistral for French and Polish in separate articles (French, Polish).

How we measured

One English prompt of 34 tokens was translated into each language, starting from machine translation that was then checked, keeping the meaning and tone the same. Tokens were counted with OpenAI's published tokenizers (cl100k_base and o200k_base). One prompt is a small sample, so longer texts and different topics will give somewhat different ratios, but the large gaps between scripts are far bigger than that variation. Full details and per-language articles are in our 41-language study.

Paying less in any language

The easiest saving is still to write instructions in English and ask for the answer in your language. See How to cut the token cost of non-English prompts, or paste your prompt into the token counter and press Save tokens to see the difference.