TokenSave Data · Dataset

How many tokens each language needs on Claude, Gemini and GPT

The same Korean prompt needs 89 tokens on Claude Opus/Sonnet 5.5, 49 on GPT and 48 on Gemini. Across 40 non-English languages, the median extra cost versus English is +41% on Gemini, +52% on GPT and +88% on Claude 5.5. The widest gap is Kannada: +577% on Claude 5.5 against +71% on Gemini.

Measured 2026-10-17 · 41 languages · 4 tokenizers · License CC BY 4.0 · Companion to the GPT-only dataset

DOI 10.5281/zenodo.23428350 · Also on Hugging Face · Kaggle · GitHub

Key findings

• English itself costs more on Claude 5.5. The English prompt is 47 tokens on Claude Opus/Sonnet 5.5, 35 on Claude Haiku 4.5, and 34 on both GPT and Gemini.
• Claude's newer tokenizer changed Latin-script text only. Compared with Haiku 4.5, Claude 5.5 needs 34% more tokens for English and more for every other Latin-script language, while Korean, Japanese, Chinese, Cyrillic, Arabic, Hebrew, Greek, Thai and the Indic languages have exactly the same counts. That is why the gap versus English looks smaller on Claude 5.5 (Korean +89%) than on Haiku 4.5 (Korean +154%).
• Indic languages are the costliest on Claude. Hindi is +134% on Claude 5.5 against +50% on GPT and +21% on Gemini.
• Gemini has the smallest gap overall. Simplified Chinese costs the same as English on Gemini (±0%).
• Models from the same family share counts. Claude Opus 5.5 and Sonnet 5.5 gave identical counts for all 41 languages; Gemini 3.8 Flash and 3.1 Pro matched on every language we re-checked.

All 41 languages

Tokens for the same prompt, with the difference versus English in brackets. Sorted by Claude 5.5.

Method

• Same prompt and translations as our GPT dataset: “Please summarize the customer email below in three bullet points and suggest a polite reply. The customer says the order arrived two days late and one item was missing from the box.” in 41 languages. Every translated prompt is in the CSV.
• Claude: each prompt was sent through Claude Code (claude -p) with a fixed one-line system prompt, and the input tokens it reported (uncached + cache read + cache write) were reduced by the same run with a one-character message. Re-running with a different system prompt, a different baseline character and a different order gave identical counts.
• Gemini: the prompt token counter in Google AI Studio, which adds one start token (“a” shows 2, “a a a” shows 4); we subtracted it. Measured on Gemini 3.8 Flash and re-checked on Gemini 3.1 Pro.
• GPT: OpenAI's tiktoken with o200k_base.
• “vs English” = tokens in the language ÷ English tokens on the same tokenizer − 1, rounded to a whole percent.

Limitations

• One prompt. With other text the ratios move by roughly ±10–20 percentage points; the ranking is more stable than the exact numbers.
• Claude counts come from usage reports rather than a dedicated count endpoint, so each value can be off by about ±2 tokens (under 5 percentage points against English).
• Input text only; output length is not measured. Prices differ by model, so more tokens does not always mean a higher bill. Use the token calculator to compare costs.

How to cite

Free to use, quote and republish with attribution under CC BY 4.0. Please link to this page or the DOI.

Embed the chart

Related

• GPT-only dataset with the older cl100k tokenizer
• Our Claude Korean test (input and output, in Korean)
• Count tokens for your own text

Questions or corrections: contact@jonhisking.com