Measured notes. Numbers below were generated on 2026-11-05 with tiktoken 0.14.0, tokenizers 0.23.3, transformers 5.22.1 and sentencepiece 0.2.3. The harness and the exact samples are in Appendix B.
- Why the same sentence costs a different number of tokens depending on the script.
- Which of the resulting problems are cost, and which are correctness bugs.
- How to measure a tokenizer before committing to it.
- What can actually be changed, and what only looks like a fix.
- Which library to reach for, and what each one will not do for you.
The short version: BPE operates on UTF-8 bytes, and the merge table is allocated by corpus frequency. A language that is 2–4 bytes per character and rare in the training mix gets few merges, so its text fragments into sub-words, single characters, or raw bytes. Alphabetic scripts with low corpus share (Amharic, Georgian, Armenian, Tamil, Bengali, Thai) are worse off than CJK, which is 3 bytes per character but extremely frequent and has a small, stable inventory. The dominant variable is not the encoding; it is the vocabulary allocation.
- A non-Latin letter is 2–4 byte-symbols before any merge. English is 1.
- Tokenizers trained mostly on English spend most of their vocabulary on Latin
byte sequences. In
cl100k_base, 89.6% of entries are Latin-script and 15 entries are Devanagari. - In the measured samples,
cl100k_baseneeded 36 tokens for a 64-character Russian sentence and 10 for a 44-character English one;o200k_baseneeded 15 for the same Russian sentence. Same text, same tokenizer family, 2.4× difference traceable to vocabulary size and training mix. - The worst measured case was
cl100k_baseon a 26-character Amharic sentence: 65 tokens, 60 of them raw bytes, 43 token boundaries inside grapheme clusters. - Cluster boundaries, normalization, and
[UNK]are correctness problems. Token count is a cost problem. They need different fixes. - "Byte fallback" preserves round-trip and destroys semantics.
[UNK]destroys information outright. Neither is support for a script. - Measure fertility, bytes per token, raw-byte/
[UNK]rate, and grapheme-cluster break rate on your own corpus. A single sentence is an illustration, not a benchmark.
A tokenizer sits at the intersection of four units that are easy to conflate:
| Unit | Example | Size |
|---|---|---|
| byte | 0xD0 0x9C |
1 byte each |
| Unicode code point | М U+041C |
2 bytes in UTF-8 |
| grapheme cluster (UAX #29) | e + U+0301 |
2 code points, 3 bytes, one visible glyph |
| token | whatever the vocabulary holds | 1 byte to several characters |
UTF-8 length is fixed by code point range:
| Code points | UTF-8 bytes | Examples |
|---|---|---|
| U+0000–U+007F | 1 | ASCII, Latin without diacritics |
| U+0080–U+07FF | 2 | Latin-1 letters, Greek, Cyrillic, Hebrew, Arabic |
| U+0800–U+FFFF | 3 | Devanagari, Bengali, Tamil, Thai, Ethiopic, Georgian, Armenian, Han, Kana, Hangul |
| U+10000–U+10FFFF | 4 | emoji, rare scripts, some CJK extensions |
Two strings can be equal as text and different as bytes:
é U+00E9 2 bytes (NFC, precomposed)
é e + U+0301 3 bytes (NFD, decomposed)
The measured tokenizers disagree about this. With cl100k_base:
precomposed "é" 1 token : é
decomposed "é" 2 tokens: e · ́
With XLM-R and Qwen2.5, both forms become one token, because their normalizers compose the sequence before segmenting:
XLM-R decomposed "é" 1 token : é
Qwen decomposed "é" 1 token : é
One emoji can be even more extreme. The family emoji is one grapheme cluster, seven code points, 25 UTF-8 bytes, and up to 19 tokens:
👨👩👧👦
cl100k 18 tokens: \xf0\x9f · \x91 · \xa8 · \xe2\x80 · \x8d · ... (11 raw bytes)
Qwen2.5 10 tokens: 👨 · \xe2\x80 · \x8d · 👩 · \xe2\x80 · \x8d · 👧 · \xe2\x80 · \x8d · 👦
So "one character" is not a useful unit anywhere in a tokenizer pipeline. The only safe unit for display and streaming is the grapheme cluster; the only safe unit for storage is bytes or code points with a declared normalization form.
flowchart LR
A[Input text] --> B[Normalizer<br/>NFC / NFKC / NMT / none]
B --> C[Pre-tokenizer<br/>regex, whitespace, script split]
C --> D[Model<br/>BPE merges / Unigram / WordPiece]
D --> E[Token IDs]
E --> F[Decoder<br/>ByteLevel / Metaspace / ByteFallback]
F --> G[Text]
Every stage can change the token count and the identity of the text.
The GPT-2 family (gpt2, r50k_base, p50k_base, cl100k_base,
o200k_base) starts from 256 byte symbols and learns pair merges:
- Count adjacent symbol pairs over the corpus.
- Merge the most frequent pair into a new symbol.
- Repeat until the vocabulary budget is exhausted.
Because the base alphabet is all 256 byte values, every input is representable and decoding is lossless. But a symbol is a byte sequence, not a character. A merge table learned on English has almost nothing to say about the byte sequences that make up Amharic, so those characters remain one, two, or three symbols each — and they may be split at any byte boundary, including inside a UTF-8 encoding of a single character.
That is exactly what cl100k_base does to Greek and Hebrew. It emits
\xce and \x97 as separate tokens for Η, and י\xd7 and \xaa for י:
Greek "Η" cl100k: \xce · \x97
Hebrew "י" cl100k: י\xd7 · \xaa
Each token boundary is inside a code point, hence inside a grapheme cluster.
SentencePiece can train BPE or a unigram language model over raw text. It
supports byte_fallback: symbols outside the vocabulary are emitted as
<0xNN> tokens, which decode back to the original bytes. It also adds a
dummy prefix space by default, which is why a first token often looks like
␣The, and it normalizes according to the model's normalization_rule_name
(nmt_nfkc by default for many models).
WordPiece (BERT, mBERT) uses greedy longest-match over a vocabulary and marks
continuations with ##. It has [UNK] rather than byte fallback: a character
outside the vocabulary is replaced, not encoded. In the measurements below,
mBERT turned an entire Amharic sentence into [UNK] tokens and kept one
content token (the final punctuation).
GPT-2's pre-tokenizer is a regex that groups letters and numbers:
's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+
\p{L}+ treats an entire run of letters with no whitespace as one "word". For
English that is the right unit. For Thai, Lao, Khmer, Burmese and Tibetan,
where spaces do not delimit words, the whole clause becomes one unit and BPE
then splits it arbitrarily. For scripts with combining marks, the regex does
not keep marks attached to their base in any linguistically meaningful way;
that is left to the merge table.
cl100k_base uses a similar pattern with \p{N}{1,3} for digits. The
practical consequence is that a Thai sentence is tokenized with no knowledge
of Thai word boundaries, and a numeral like 12345 is chunked by a rule that
has nothing to do with the language.
Vocabulary size alone says little. What matters is how many entries serve each script. I classified every vocabulary entry by the majority script of its alphabetic characters; entries that are byte fragments, punctuation, digits or markers are left unclassified, so these are lower bounds and the columns do not sum to 100%.
| encoding | vocab | Latin | Cyrillic | Arabic | Devanagari | Hebrew | Greek | Han | Kana | Hangul | Thai | Ethiopic | Georgian |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| cl100k | 100256 | 89823 (89.6%) | 736 (0.7%) | 98 (0.1%) | 15 (0.0%) | 22 (0.0%) | 44 (0.0%) | 867 (0.9%) | 163 (0.2%) | 299 (0.3%) | 46 (0.0%) | 0 | 0 |
| o200k | 199998 | 134776 (67.4%) | 14212 (7.1%) | 7965 (4.0%) | 3900 (2.0%) | 2361 (1.2%) | 1507 (0.8%) | 7506 (3.8%) | 754 (0.4%) | 2373 (1.2%) | 1539 (0.8%) | 0 | 2200 (1.1%) |
| gpt2 | 50256 | 47003 (93.5%) | 18 (0.0%) | 19 (0.0%) | 0 | 11 (0.0%) | 22 (0.0%) | 50 (0.1%) | 132 (0.3%) | 2 (0.0%) | 0 | 0 | 0 |
| XLM-R | 250002 | 109981 (44.0%) | 31670 (12.7%) | 14423 (5.8%) | 7515 (3.0%) | 5141 (2.1%) | 5174 (2.1%) | 17773 (7.1%) | 2788 (1.1%) | 5413 (2.2%) | 4299 (1.7%) | 2951 (1.2%) | 3770 (1.5%) |
| mBERT | 119547 | 66542 (55.7%) | 13795 (11.5%) | 4680 (3.9%) | 1740 (1.5%) | 2450 (2.0%) | 1564 (1.3%) | 13548 (11.3%) | 1329 (1.1%) | 3273 (2.7%) | 320 (0.3%) | 0 | 705 (0.6%) |
| Mistral | 32000 | 25980 (81.2%) | 1731 (5.4%) | 57 (0.2%) | 32 (0.1%) | 27 (0.1%) | 75 (0.2%) | 1459 (4.6%) | 133 (0.4%) | 346 (1.1%) | 44 (0.1%) | 8 (0.0%) | 29 (0.1%) |
| Qwen2.5 | 151665 | 94494 (62.3%) | 4144 (2.7%) | 3979 (2.6%) | 56 (0.0%) | 3159 (2.1%) | 235 (0.2%) | 25469 (16.8%) | 1776 (1.2%) | 3586 (2.4%) | 2550 (1.7%) | 112 (0.1%) | 35 (0.0%) |
Three things stand out:
cl100k_baseis an English tokenizer with guests. 89.6% Latin, 15 Devanagari entries, zero Ethiopic and zero Georgian. Greek has 44 entries for a 24-letter alphabet; that is enough to cover some frequent bigrams but not the language.o200k_baseis the same architecture with a bigger, more balanced vocabulary. It has 19× the Cyrillic entries, 260 times the Devanagari entries, and 2,200 Georgian entries. The measured cost drops accordingly (Russian 36 → 15 tokens, Georgian 73 → 12).- Multilingual models make a different trade. XLM-R's 250k vocabulary is broad but still 44% Latin; mBERT's 120k is 56% Latin; Mistral's 32k is 81% Latin and has 32 Devanagari entries. Qwen2.5 spends 16.8% of its vocabulary on Han and only 56 entries on Devanagari, which matches its measured behaviour: excellent Chinese, poor Hindi.
o200k_base has zero Ethiopic entries by this classification, yet it improves
on cl100k for Amharic (65 → 48 tokens) because it has more byte-sequence
merges that happen to cover Ethiopic syllables. Vocabulary share is a useful
predictor, not a law.
Notation in the blocks below: ␣ marks a leading space inside a token; \xNN
is a raw byte that is not valid UTF-8 on its own. Token counts exclude
BOS/EOS/CLS/SEP/PAD; [UNK]/<unk> is reported separately. Cluster-break
counts are only quoted for tokenizers whose pieces reconstruct the source bytes
exactly (all byte-level models, XLM-R and Mistral); WordPiece's pre-tokenizer
drops whitespace around punctuation, so mBERT is excluded from that metric.
"The quick brown fox jumps over the lazy dog." 44 chars, 44 bytes, 9 words
cl100k 10 tok: The · ␣quick · ␣brown · ␣fox · ␣jumps · ␣over · ␣the · ␣lazy · ␣dog · .
XLM-R 13 tok: The · ␣quick · ␣brown · ␣ · fox · ␣jump · s · ␣over · ␣the · ␣la · zy · ␣dog · .
English is the case every tokenizer is optimized for: roughly 1.1 tokens per word, no cluster breaks, no raw bytes. It is the denominator for every other comparison in this document.
"L'élève étudie la linguistique comparée." 40 chars, 44 bytes, 5 words
cl100k 13 tok: L · ' · él · ève · ␣ét · ud · ie · ␣la · ␣lingu · istique · ␣compar · ée · .
o200k 12 tok: L · 'él · ève · ␣ét · ud · ie · ␣la · ␣lingu · istique · ␣compar · ée · .
French is close to English: 2.6 tokens per word for cl100k, 2.4 for o200k.
The precomposed é is one token; the cost comes from the apostrophe and the
longer words, not from Unicode.
Vietnamese is the hard Latin case. It stacks tone marks and vowel modifiers on a Latin base, so each syllable is 2–3 bytes, and the byte sequences are far less frequent in an English corpus:
"Học sinh học tiếng Việt mỗi ngày." 33 chars, 44 bytes, 7 words
cl100k 16 tok: H · ọc · ␣sinh · ␣h · ọc · ␣ti · ế · ng · ␣Vi · ệ · t · ␣m · ỗi · ␣ng · ày · .
gpt2 31 tok, 15 raw-byte, 10 cluster breaks:
H · \xe1 · \xbb · \x8d · c · ␣sin · h · ... (the bytes of "ọ", "ế", "ệ", "ỗ" split)
XLM-R 8 tok: Học · ␣sinh · ␣học · ␣tiếng · ␣Việt · ␣mỗi · ␣ngày · .
The same text is 2.3 tokens per word in cl100k, 1.3 in o200k, and 1.1 in
XLM-R, which has Vietnamese in its training mix. gpt2 is 4.4 tokens per word
and splits 15 of 31 tokens into raw bytes.
The lesson is not "Latin is safe". It is "Latin script with a low-frequency diacritic pattern behaves like any other low-resource script".
"Η γλώσσα είναι το σπίτι του πολιτισμού." 39 chars, 71 bytes, 7 words
cl100k 34 tok, 6 raw-byte, 3 cluster breaks:
\xce · \x97 · ␣γ · λ · \xcf · \x8e · σ · σ · α · ␣ε · ί · ν · αι · ␣τ · ο · ...
o200k 12 tok: Η · ␣γ · λώ · σσα · ␣είναι · ␣το · ␣σπίτι · ␣του · ␣πολι · τισ · μού · .
XLM-R 8 tok: Η · ␣γλώσσα · ␣είναι · ␣το · ␣σπίτι · ␣του · ␣πολιτισμού · .
Mistral 39 tok: Η · ␣ · γ · λ · ώ · σ · σ · α · ␣ · ε · ...
Greek has 24 letters and is 2 bytes per character. That does not save it:
cl100k has 44 Greek entries, so it tokenizes close to one character per
token, and occasionally splits a character's bytes. o200k's 1,507 Greek
entries bring it to 1.7 tokens per word. Mistral's 32k vocabulary has 75 Greek
entries, so it is one character per token — 39 tokens for 7 words.
A script with a small alphabet still needs vocabulary entries for its common bigrams and suffixes. Alphabet size is not the predictor; merge allocation is.
"Москва — столица России, а Санкт-Петербург — культурная столица."
64 chars, 119 bytes, 9 words
cl100k 36 tok: М · ос · к · ва · ␣— · ␣ст · ол · и · ца · ␣Р · ос · с · ии · , · ...
o200k 15 tok: Москва · ␣— · ␣столи · ца · ␣России · , · ␣а · ␣Санкт · -Петербург · ...
XLM-R 14 tok: Москва · ␣— · ␣столица · ␣России · , · ␣а · ␣Санкт · - · Петербург · ...
Mistral 24 tok: Моск · ва · ␣— · ␣сто · ли · ца · ␣России · , · ␣а · ␣Сан · кт · ...
Cyrillic is the best-documented "middle" case: 2 bytes per character, highly
inflected, and present in most multilingual corpora. The measured spread is
2.4× between cl100k (4.0 tokens/word) and o200k (1.7 tokens/word), and 4.5×
against gpt2 (7.6 tokens/word, 18 raw bytes, 12 cluster breaks). All four
tokenizers here use whole code points; the raw bytes in gpt2 are the
exception that proves the rule — they come from byte-level BPE, not from
Cyrillic itself.
Both are 2 bytes per character, written right-to-left, and have no case. The tokenizer sees logical order; bidi reordering is a display concern. That means bidi control characters (U+202A–U+202E, U+2066–U+2069) should be removed before tokenization, not counted as text. They are also a prompt-injection vector, because they can make a string read differently from what it is.
"השפה העברית היא שפה שמית עתיקה." 31 chars, 56 bytes, 6 words
cl100k 29 tok, 9 raw-byte, 7 cluster breaks:
ה · ש · \xd7 · \xa4 · ה · ␣ה · ע · ב · ר · י\xd7 · \xaa · ␣ה · י\xd7 · \x90 · ...
o200k 13 tok: הש · פה · ␣הע · בר · ית · ␣היא · ␣ש · פה · ␣שמ · ית · ␣עת · יקה · .
XLM-R 12 tok: ה · שפה · ␣העברית · ␣היא · ␣ · שפה · ␣שמ · ית · ␣ · עת · יקה · .
"اللغة العربية من أقدم اللغات السامية." 37 chars, 68 bytes, 6 words
cl100k 23 tok: ال · ل · غ · ة · ␣ال · ع · ر · ب · ية · ␣من · ␣أ · ق · د · م · ...
o200k 11 tok: ال · لغة · ␣العربية · ␣من · ␣أ · قدم · ␣الل · غات · ␣الس · امية · .
XLM-R 9 tok: اللغة · ␣العربية · ␣من · ␣أ · قدم · ␣اللغات · ␣الس · امية · .
Hebrew is the more expensive of the two in the measured tokenizers: 4.8
tokens/word in cl100k versus 3.8 for Arabic. Both improve by 2× or more in
o200k. Mistral's small vocabulary is close to one character per token for
both (31 tokens for the Hebrew sentence, 29 for the Arabic one).
Adding short vowels and shadda changes the picture: the marks are separate combining code points, and every tokenizer in the measurement splits at least some of them from their base letter.
"اللُّغَةُ العَرَبِيَّةُ جَمِيلَةٌ." 34 chars, 65 bytes, 3 words
cl100k 30 tok, 15 cluster breaks: ال · ل · ُ · ّ · غ · َ · ة · ُ · ␣ال · ع · َ · ر · ...
o200k 22 tok, 14 cluster breaks: الل · ُ · ّ · غ · َ · ة · ُ · ␣الع · َر · َب · ِي · ...
XLM-R 25 tok, 12 cluster breaks: الل · ُّ · غ · َ · ة · ُ · ␣الع · َ · ر · َ · ب · ِ · ...
The marks inflate the count for the same content (30 tokens for 34 characters,
against 23 for 37 unvocalized characters in cl100k), and around half of all
token boundaries now fall inside a grapheme cluster. A streamer that emits tokens as
they arrive will render a combining mark on the wrong base, or a bare mark.
This is the same class of bug as splitting a ZWJ emoji sequence.
The same applies to Hebrew niqqud (U+05B0–U+05BC) and to Indic vowel signs. Normalization does not fix it: NFC does not compose Arabic letter + haraka into a single code point, because Unicode does not define such compositions.
"हिन्दी भारत की सबसे अधिक बोली जाने वाली भाषा है।"
48 chars, 126 bytes, 10 words
cl100k 49 tok, 17 raw-byte, 33 cluster breaks:
ह · ि\xe0\xa4 · \xa8 · ्\xe0\xa4 · \xa6 · ी · ␣\xe0\xa4 · \xad · ा\xe0\xa4 · \xb0 · त · ...
o200k 14 tok, 2 cluster breaks:
ह · िन्द · ी · ␣भारत · ␣की · ␣सबसे · ␣अधिक · ␣बो · ली · ␣जाने · ␣वाली · ␣भाषा · ␣है · ।
XLM-R 11 tok: हिन्दी · ␣भारत · ␣की · ␣सबसे · ␣अधिक · ␣बोली · ␣जाने · ␣वाली · ␣भाषा · ␣है · ।
Qwen2.5 46 tok, 30 cluster breaks: (not shown; 56 Devanagari entries in the vocabulary)
Devanagari is 3 bytes per character, and its vowel signs (matras) are separate
combining code points. The word हिन्दी is six code points in two grapheme
clusters (the virama joins न्द); cl100k gives it six tokens, with
boundaries inside the matra ि and inside the UTF-8 encoding of न. The conjunct क्ष (क + virama +
ष) is three code points and one cluster; tokenizers that do not know it will
split it.
o200k is 3.5× cheaper than cl100k here and cuts cluster breaks from 33 to
2. XLM-R is better still. Qwen2.5, despite being a modern multilingual model
with a 151k vocabulary, is nearly as bad as cl100k on Hindi: it has 56
Devanagari entries. Vocabulary share predicts this exactly.
The Dravidian and Eastern Indic scripts show the same pattern with more agglutination and longer words.
"বাংলা ভাষা দক্ষিণ এশিয়ার একটি গুরুত্বপূর্ণ ভাষা।"
49 chars, 135 bytes, 7 words
cl100k 62 tok, 27 raw-byte, 44 cluster breaks
o200k 14 tok, 5 cluster breaks: বাংল · া · ␣ভাষ · া · ␣দক্ষিণ · ␣এ · শ · িয় · ার · ...
XLM-R 9 tok: বাংলা · ␣ভাষা · ␣দক্ষিণ · ␣এশিয়া · র · ␣একটি · ␣গুরুত্বপূর্ণ · ␣ভাষা · ।
"தமிழ் மொழி மிகப் பழமையான மொழிகளில் ஒன்றாகும்."
45 chars, 123 bytes, 6 words
cl100k 63 tok, 28 raw-byte, 44 cluster breaks
o200k 15 tok, 6 cluster breaks
XLM-R 12 tok, 2 cluster breaks
gpt2 123 tok, 117 raw-byte, 94 cluster breaks
Bengali is 8.9 tokens per word in cl100k and 1.3 in XLM-R. Tamil is 10.5
tokens per word in cl100k and 2.0 in XLM-R; gpt2 spends more tokens than
the text has bytes (123 tokens for 123 bytes, 117 of them raw bytes). The
cl100k vocabulary has 15 Devanagari entries and effectively nothing for
Tamil; the model is transliterating the script into bytes.
"ภาษาไทยเป็นภาษาที่มีเสียงวรรณยุกต์" 34 chars, 102 bytes, 1 whitespace-delimited run
cl100k 36 tok, 4 raw-byte, 10 cluster breaks:
\xe0\xb8 · \xa0 · า · \xe0\xb8 · \xa9 · า · ไ · ท · ย · เ · ป · ็ · น · ...
o200k 13 tok, 1 cluster break: ภาษา · ไทย · เป็น · ภ · าษ · าที่ · มี · เสียง · ว · รรณ · ...
XLM-R 9 tok: ภาษาไทย · เป็น · ภาษา · ที่มี · เสียง · วรรณ · ยุ · ก · ต์
Qwen2.5 17 tok, 6 cluster breaks
There are no spaces between words, so "tokens per word" is undefined here; the
whitespace pre-tokenizer sees one run of 34 characters. cl100k costs 1.06
tokens per character, about 4.6× the English per-character cost, and 10 of its
34 token boundaries fall inside a grapheme cluster (Thai vowel and tone marks
stack above and below the base). XLM-R, which has Thai coverage and a
word-aware SentencePiece model, gets the whole sentence into 9 tokens.
If you need word-level behaviour for Thai, Lao, Khmer, Burmese or Tibetan, you need a dictionary or ICU word-break segmenter before BPE. A regex over whitespace cannot supply what the script does not have.
"አማርኛ የኢትዮጵያ የመንግስት ቋንቋ ነው።" 26 chars, 70 bytes, 5 words
cl100k 65 tok, 60 raw-byte, 43 cluster breaks: (2.5 tokens per character)
o200k 48 tok, 22 raw-byte, 22 cluster breaks
XLM-R 5 tok: አማርኛ · ␣የኢትዮጵያ · ␣የመንግስት · ␣ቋንቋ · ␣ነው።
mBERT 0 content tokens, 6 [UNK] (the sentence is discarded)
Mistral 52 tok, 39 raw-byte, 26 cluster breaks
Ethiopic is a syllabary: each base character encodes a consonant–vowel
syllable, with several hundred signs in the U+1200 block. It is 3 bytes per
character and rare in English-dominated corpora, which is the worst
combination. cl100k emits 60 raw bytes out of 65 tokens — it is not
tokenizing Amharic, it is storing its UTF-8. mBERT replaces every content
character with [UNK], so a downstream model sees a sentence of six unknown
tokens.
This is the clearest case where "the tokenizer supports Unicode" is true in a trivial sense (no crash, round-trip via bytes) and false in every useful sense.
"ქართული ენა ქართველების მშობლიური ენაა."
39 chars, 107 bytes, 5 words
cl100k 73 tok, 34 raw-byte, 34 cluster breaks (14.6 tokens/word)
o200k 12 tok: ქართული · ␣ე · ნა · ␣ქართველ · ების · ␣მშ · ობლ · იური · ␣ე · ნა · ა · .
XLM-R 10 tok: ქართული · ␣ენა · ␣ქართველები · ს · ␣ · მშობლ · იური · ␣ენა · ა · .
"Հայերենը հնդեվրոպական լեզու է։" 30 chars, 57 bytes, 4 words
cl100k 57 tok, 54 raw-byte, 27 cluster breaks (14.2 tokens/word)
o200k 13 tok, 1 raw-byte
XLM-R 11 tok
Georgian and Armenian have their own alphabets, 3 bytes per character, and
small corpora. cl100k has zero Georgian and zero Armenian entries, so both
come out near one token per byte: 14 tokens per word. o200k
fixes Georgian almost completely (2,200 entries) and Armenian mostly.
These are the cases that most clearly separate "large vocabulary" from
"multilingual". o200k did not add a Georgian language model; it added 2,200
merge entries, and that was enough to bring the cost from 73 tokens to 12.
CJK is often described as expensive because each character is 3 bytes. In practice it is the best-served non-Latin family, because the character inventory is bounded and extremely frequent.
"中文是世界上最古老的语言之一。" 15 chars, 45 bytes
cl100k 17 tok, 2 raw-byte: 中 · 文 · 是 · \xe4\xb8 · \x96 · 界 · 上 · 最 · ...
o200k 11 tok: 中文 · 是 · 世界 · 上 · 最 · 古 · 老 · 的 · 语言 · 之一 · 。
Qwen2.5 7 tok: 中文 · 是 · 世界上最 · 古老 · 的语言 · 之一 · 。
"日本語は文法が英語と大きく異なります。" 19 chars, 57 bytes
cl100k 21 tok, 3 raw-byte
o200k 16 tok
Qwen2.5 14 tok
XLM-R 10 tok: 日本語 · は · 文 · 法 · が · 英語 · と · 大きく · 異なります · 。
Korean is instructive because Hangul is alphabetic in structure (jamo) and 3 bytes per syllable in its precomposed form:
"한국어는 교착어로서 조사가 발달했다." 20 chars, 52 bytes, 4 words
cl100k 23 tok, 7 raw-byte, 7 cluster breaks (5.8 tokens/word)
o200k 14 tok (3.5/word)
XLM-R 11 tok (2.8/word)
gpt2 46 tok, 38 raw-byte (11.5/word)
cl100k splits some Hangul syllables into partial UTF-8; o200k and XLM-R do
not. If the text is decomposed into jamo (NFD), the cost rises further unless
the normalizer composes it first.
CJK is not the problem case. Alphabetic scripts with low corpus share and combining marks are.
"བོད་ཡིག་ནི་བོད་ཀྱི་ཡིག་སྐད་ཡིན།" 31 chars, 93 bytes, 1 whitespace run
cl100k 64 tok, 35 raw-byte, 42 cluster breaks
o200k 48 tok, 24 raw-byte, 33 cluster breaks
XLM-R 3 content tokens, 4 [UNK]
mBERT 16 content tokens, 5 [UNK]
Mistral 45 tok, 21 raw-byte, 23 cluster breaks
"ᏣᎳᎩ ᎦᏬᏂᎯᏍᏗ ᎠᏂᏴᏫ ᎤᏤᎵᎢ." 21 chars, 55 bytes, 4 words
cl100k 52 tok, 48 raw-byte, 34 cluster breaks
o200k 55 tok, 51 raw-byte, 34 cluster breaks
XLM-R 4 content tokens, 4 [UNK]
mBERT 1 content token (the period), 4 [UNK]
Qwen2.5 33 tok, 15 cluster breaks
Two different failure modes:
- Byte fallback (the GPT-2 family, Mistral here) preserves the bytes and therefore the text, at 2–3 tokens per character. The model sees a sequence with no linguistic units.
[UNK](XLM-R, mBERT) loses the information. A downstream model cannot recover the word from<unk>, and any token-level statistic is meaningless.
Neither is "support". The difference matters for what you can do next: with
byte fallback you can still round-trip, search and display the text; with
[UNK] you cannot.
Cherokee is also a reminder that a larger vocabulary does not guarantee
improvement: o200k is slightly worse than cl100k here (55 vs 52 tokens,
51 vs 48 raw bytes), while Qwen2.5 is better (33 tokens). Vocabulary
composition is not monotone in size.
👨👩👧👦 7 code points, 25 bytes, 1 grapheme cluster
cl100k 18 tok, 11 raw-byte, 17 cluster breaks (100% of interior boundaries)
o200k 11 tok, 4 raw-byte, 10 cluster breaks
Qwen2.5 10 tok, 9 cluster breaks
XLM-R 7 tok, 6 cluster breaks
👍🏽 2 code points, 8 bytes, 1 cluster
cl100k 6 tok
Qwen2.5 2 tok
🏳️🌈 4 code points, 14 bytes, 1 cluster
cl100k 9 tok
Qwen2.5 5 tok
The family emoji is the strongest single example in this document: one visible glyph, 18 tokens, and every interior token boundary inside the cluster. A token-level streaming loop will paint the family one person at a time, with dangling ZWJ characters between them. A cursor movement or truncation routine that counts tokens will cut the glyph in half.
Byte cost is not the whole story. Turkish is mostly 1 byte per character in Latin script, yet its morphology produces long words and many suffix chains:
"Türkçe, eklemeli bir dil olduğu için sözcükler uzayabilir."
58 chars, 64 bytes, 8 words
cl100k 22 tok (2.8/word): T · ür · k · ç · e · , · ␣ek · lem · eli · ␣bir · ␣dil · ...
o200k 19 tok (2.4/word)
XLM-R 15 tok (1.9/word): Türkçe · , · ␣ek · leme · li · ␣bir · ␣dil · ␣olduğu · ...
A single Finnish word can be many tokens:
"talossanikin" ("in my house too") 12 chars, 12 bytes
cl100k 5 tok: tal · oss · an · ik · in
o200k 4 tok: tal · oss · an · ikin
XLM-R 4 tok: tal · ossa · ni · kin
Finnish, Hungarian, Turkish and Swahili are 2.8–3.2 tokens per word in
cl100k, versus 1.1 for English. The cost comes from suffix boundaries, not
from UTF-8. A vocabulary trained with these languages gets the suffixes as
units and the cost drops toward 1.8–2.6.
- Context window. The same information needs 2–6× the tokens for several
scripts in the measured tokenizers. Against the English sample, the measured
cl100ksentences cost 3.6× for Russian, 4.9× for Hindi and 6.5× for Amharic, so an 8k-token budget holds proportionally less of each. - Money and latency. Both scale with tokens on most APIs. Tokenizers that fragment a script make that script more expensive per unit of meaning.
- Attention dilution. Longer sequences for the same content mean weaker per-token signal and more chances for the model to lose a fact. This is a modelling claim, not something the measurements here prove; the token-count disparity is measured, the quality effect is inferred.
Measured interior-boundary rates for the worst samples:
| Sample | cl100k | o200k | XLM-R | Mistral |
|---|---|---|---|---|
| Arabic with harakat | 52% | 67% | 50% | 47% |
| Hindi | 69% | 15% | 0% | 36% |
| Bengali | 72% | 38% | 0% | 50% |
| Tamil | 71% | 43% | 18% | 52% |
| Thai | 29% | 8% | 0% | 21% |
| Amharic | 67% | 47% | 0% | 51% |
| Emoji ZWJ | 100% | 100% | 100% | 100% |
Consequences that are easy to ship as bugs:
- Streaming. Emit only at grapheme-cluster boundaries (UAX #29), buffering incomplete clusters. Otherwise a combining mark arrives before its base, or a ZWJ sequence is painted piecewise.
- Truncation. Cutting at N tokens can cut a cluster in half, leaving a dangling combining mark. Cut at token and cluster boundaries.
- Constrained decoding / token healing. If a grammar constrains the next characters, the token boundary may force a partial code point. Token healing or byte-level constraints are needed.
- Editing and search. A cursor or highlight that indexes tokens or code points will move through one visible glyph multiple times.
- Diffing. Token-level diffs of two languages with different fertility are not comparable, and a diff that splits inside a cluster renders badly.
- Same text, different tokens. NFC and NFD forms of
éare 1 and 2 tokens incl100k. Arabic with and without harakat, or with different alef forms (أvsا), tokenizes differently and compares unequal as strings. - NFKC is lossy. It folds full-width forms, ligatures, superscripts and some presentation forms. It is the right choice when compatibility matters and the wrong choice when the exact source text matters.
- Do not normalize twice. If the tokenizer normalizes and your code also does, offsets and equality checks may disagree. Pick one canonical form at the boundary and record it.
- Security. Bidi controls, zero-width characters and Unicode tag characters can hide text. Strip them before tokenizing untrusted input, and keep the visible text separate from the wire text.
[UNK]makes the model blind to the text and makes any per-token metric wrong. If a language is in scope and the[UNK]rate is non-trivial, the tokenizer does not cover it.- Byte fallback preserves the text but spends 1 token per byte. A 90% raw-byte rate means the vocabulary has no linguistic units for the script.
- Both can be present in the same model: some SentencePiece models have byte
fallback and still emit
<unk>for characters the normalizer maps away.
Perplexity is per token. A model with a 4× higher fertility on Language A will look better or worse than it is when compared to Language B on token-level metrics. Compare bits per character or bits per byte, or normalize by fertility. BLEU and other surface metrics computed on token sequences inherit the same problem.
For a corpus per language:
| Metric | Definition | What it tells you |
|---|---|---|
| fertility | tokens / whitespace word | cost per word for spaced scripts |
| tokens per character | tokens / code point | script-level fragmentation |
| bytes per token | UTF-8 bytes / token | compression; often around 4 for natural English text |
| raw-byte rate | raw-byte tokens / tokens | how much of the text is stored as bytes |
| UNK rate | [UNK] tokens / tokens |
information loss |
| cluster-break rate | interior token boundaries not on a UAX #29 boundary / interior boundaries | rendering and streaming risk |
| parity ratio | tokens for a parallel text / tokens for its English version | fairness across languages |
The last one requires a parallel corpus. Single translated sentences are illustrations; use a few thousand sentences per language before drawing conclusions.
import regex, tiktoken
from transformers import AutoTokenizer
# byte-level model
enc = tiktoken.get_encoding("o200k_base")
ids = enc.encode("हिन्दी")
pieces = [enc.decode_single_token_bytes(i).decode("utf-8", "backslashreplace") for i in ids]
# SentencePiece / WordPiece model, with offsets
tok = AutoTokenizer.from_pretrained("xlm-roberta-base", use_fast=True)
out = tok("हिन्दी", return_offsets_mapping=True, add_special_tokens=False)
print(out.tokens(), out["offset_mapping"])Cluster boundaries, without a full Unicode library:
clusters = regex.findall(r"\X", text) # UAX #29 grapheme clusters
starts, pos = set(), 0
for c in clusters:
starts.add(pos) # pos in bytes, see note
pos += len(c.encode())If you work in bytes, encode each cluster and accumulate byte lengths; if you
work in code points, accumulate len(c). Do not mix the two. The regex
module's \X is the practical implementation of UAX #29 in Python.
- Counting special tokens.
encode()may add<s>,[CLS],[SEP]. Strip them before comparing counts. In the measurements above, they are excluded. - The dummy prefix space. SentencePiece's
add_dummy_prefixinserts a leading space and fuses it into the first token (␣The) or emits it as a separate token. Remove one leading space before computing byte offsets, or every offset after it shifts by one. - Normalization drift. If the tokenizer normalizes (NFKC, NMT) and your reference text does not, offsets reconstructed from pieces will not line up. Normalize the reference with the same rule for the comparison, or use the tokenizer's own offsets.
- Confusing offsets. Hugging Face
tokenizersoffsets index the original string in code points (a six-character Cyrillic token reports(5, 11)), not bytes. Byte-level models have no character offsets until you decode. - Decoding partial bytes. A byte-level token may not be valid UTF-8 on its
own. Decode with
errors="backslashreplace"when printing, and keep the raw bytes for measurement. - One sentence. The samples here are deliberately short and chosen to illustrate mechanisms. They are not a corpus and their ratios will not generalize exactly.
English is the baseline at ~1.1 tokens per word. For the same sentence content,
cl100k costs 4.0 tokens/word in Russian, 4.9 in Greek and Hindi, 8.9 in
Bengali, 10.5 in Tamil, 13.0 in Amharic and 14.6 in Georgian. o200k brings
those to 1.7, 1.7, 1.4, 2.0, 2.5, 9.6 and 2.4 respectively. XLM-R is close to
1–2 tokens/word for most of them, at the cost of a 250k vocabulary and a
different model family.
The full per-sample table is in Appendix A. The important reading is the shape, not the exact numbers: the spread between tokenizers on the same text is often larger than the spread between languages.
The only real fix is to allocate vocabulary and training data to the languages you care about:
- Train or extend the merge table on a corpus that has the target languages in proportion to their importance to you. A 1% share produces a 1%-quality tokenizer for that language, regardless of vocabulary size.
- Increase vocabulary size if you can afford the embedding matrix. The measured
cl100k→o200kchange is essentially this, plus a better training mix. - Prefer a tokenizer trained on the scripts you need over a larger one trained
on English. XLM-R's 250k vocabulary beats
cl100k's 100k on every non-Latin sample here.
If you must keep the model:
- Train or choose the new tokenizer.
- Expand the embedding matrix for the new tokens.
- Initialize new embeddings (mean of the old embeddings, or train them).
- Freeze or slowly unfreeze the old embeddings.
- Continue pretraining on a balanced corpus.
This is standard practice but it is a training project, not a configuration flag. It also changes the token-to-logit mapping, so any saved prompt, constraint or cached token sequence must be re-encoded.
If you control the tokenizer, split on script boundaries and keep combining
marks with their base. With Hugging Face tokenizers:
from tokenizers import Tokenizer, models, trainers, pre_tokenizers, decoders, normalizers
tok = Tokenizer(models.BPE(byte_fallback=True))
tok.normalizer = normalizers.NFC()
tok.pre_tokenizer = pre_tokenizers.Sequence([
pre_tokenizers.UnicodeScripts(), # split runs by script
pre_tokenizers.Metaspace(replacement="▁", prepend_scheme="always"),
])
tok.decoder = decoders.Sequence([decoders.Metaspace(replacement="▁", prepend_scheme="always"),
decoders.ByteFallback()])
tok.train_from_iterator(corpus, trainers.BpeTrainer(
vocab_size=64000,
special_tokens=["<unk>", "<s>", "</s>"],
))UnicodeScripts splits a mixed-script run into script-homogeneous pieces
(verified: helloПривет becomes hello and Привет), which stops one script
from stealing merges from another. It does not by itself keep Indic matras or
Arabic harakat attached; that needs a rule or a pre-segmented corpus.
For space-free scripts, replace the whitespace/regex pre-tokenizer with a word
segmenter (ICU, pythainlp, khmer-nltk, or a trained SentencePiece model
over raw text). BPE will still split within a word, but it will at least start
from linguistic units.
- NFC for storage and comparison, unless you need byte-identical source text.
- NFKC only when compatibility is intended (full-width, ligatures, presentation forms). It is lossy.
- None when the exact code points matter (security, source code, hashes, signed content). Then the tokenizer's own normalizer must be disabled, or your equality checks must use the normalized form.
Whichever you choose, do it once, at the boundary, and record which form the text is in.
Buffer token output until a complete UAX #29 cluster is available. In Go,
github.com/rivo/uniseg provides the segmenter; in Python, regex with \X;
in Rust, unicode-segmentation. This is cheap and prevents the most visible
class of multilingual rendering bug.
The same rule applies to truncation: cut at a cluster boundary, and do not split a ZWJ sequence or leave a combining mark without its base.
- Do not "fix" fertility by counting characters. Token limits, pricing and context windows are in tokens. A UI that shows "characters remaining" is lying to the user.
- Do not use byte fallback as evidence of support. It is a lossless emergency path, not a language model.
- Do not compare perplexity across languages without normalizing by bytes or characters.
- Do not add a language to the vocabulary without adding it to the training mix. A merge entry that never appears in pretraining is dead weight.
| Library | Core | Training | Normalizers | Pre-tokenizers | Byte fallback | Offsets | Best for |
|---|---|---|---|---|---|---|---|
tiktoken |
Rust, Python bindings | no | none | fixed regex per encoding | n/a (byte-level) | decode_with_offsets |
OpenAI-compatible inference, exact reproduction of GPT encodings |
tokenizers (HF) |
Rust, Python/Node bindings | BPE, Unigram, WordPiece, WordLevel | NFC/NFD/NFKC/NFKD, NMT, ByteLevel, Replace, Strip | ByteLevel, Metaspace, Whitespace, Punctuation, Digits, UnicodeScripts, Split | models.BPE(byte_fallback=True) + decoders.ByteFallback |
Encoding.offsets (code points) |
Training, custom pipelines, fast batch encoding |
sentencepiece |
C++, Python wrapper | BPE, Unigram, Word, Char | nmt_nfkc, nfkc, nfd, identity |
Metaspace-style, dummy prefix | --byte_fallback |
EncodeAsOffsetMapping |
Raw-text training, Llama/Mistral/T5-style models |
transformers.AutoTokenizer |
wrapper over the above | via train_new_from_iterator |
whatever the model uses | whatever the model uses | model-dependent | fast tokenizers only | Loading a published model's tokenizer |
tokenmonster |
C | yes | limited | n/a | no | n/a | English-centric compact vocabularies |
gigatoken |
Rust/Python | yes | limited | configurable | n/a | n/a | Very fast training/inference experiments |
The table is about capabilities, not quality. A library cannot add vocabulary coverage that the model was not trained with.
tiktoken reproduces OpenAI's encodings (gpt2, r50k_base, p50k_base,
p50k_edit, cl100k_base, o200k_base). It is byte-level BPE with no
normalizer: you get exactly the bytes you pass in.
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
ids = enc.encode("hello") # special tokens not allowed
ids = enc.encode_ordinary("hello") # never treats text as special
pieces = [enc.decode_single_token_bytes(i) for i in ids]
text = enc.decode(ids)Notes:
- The first use of a named encoding downloads its BPE file from OpenAI's blob storage; there is no training API and no normalizer.
encode()recognises special-token strings;encode_ordinary()does not. For untrusted text,encode_ordinary()is the safer default.decode_single_token_bytes()is the right primitive for measurement: a token can be a partial UTF-8 sequence.decode_with_offsets()returns the decoded text and the offsets of token boundaries in it; it is not a general character-alignment API.- The model registry (
encoding_for_model) changes over time. Pin the encoding name, not the model name, in anything reproducible.
Use it when you need OpenAI-compatible token counts or an exact reproduction of a published encoding. Do not use it to train a tokenizer.
This is the most flexible pipeline. It separates normalizer, pre-tokenizer, model, post-processor and decoder, and exposes a trainer for each model type.
from transformers import AutoTokenizer
from tokenizers import Tokenizer, models, trainers, pre_tokenizers, decoders
fast = AutoTokenizer.from_pretrained("bert-base-multilingual-cased", use_fast=True)
tok = fast.backend_tokenizer
enc = tok.encode("हिन्दी")
print(enc.tokens, enc.offsets)
tok = Tokenizer(models.BPE(unk_token="<unk>"))
tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
tok.decoder = decoders.ByteLevel()
tok.train_from_iterator(texts, trainers.BpeTrainer(vocab_size=32000,
special_tokens=["<unk>"]))Notes:
Encoding.offsetsare in code points in the Python API (verified: a six-character Cyrillic token reports(5, 11)). Do not treat them as bytes.BpeTrainer(byte_fallback=True)is silently ignored in 0.23; putbyte_fallback=Trueonmodels.BPEand adddecoders.ByteFallback().UnigramTraineraccepts abyte_fallbackkwarg that is currently ignored; check the version before relying on it.pre_tokenizers.UnicodeScripts()splits mixed-script runs by script. It is newer than the oldest releases; pin a version if you depend on it.- Batch methods (
encode_batch) and Rust-backed parallelism are the reason to choose this for training corpora.
Use it when you are training or auditing a tokenizer, or when you need a pipeline that differs from a published model's.
SentencePiece trains directly on raw text and ships a protobuf model. It is the format behind Llama, Mistral, T5, XLM-R and many multilingual models.
import sentencepiece as spm
spm.SentencePieceTrainer.train(
input="corpus.txt",
model_prefix="sp",
vocab_size=64000,
model_type="bpe", # or "unigram", "word", "char"
byte_fallback=True, # <0xNN> for anything outside the vocabulary
character_coverage=1.0, # default is 0.9995; 1.0 keeps rare scripts
normalization_rule_name="nmt_nfkc",
add_dummy_prefix=True,
)
sp = spm.SentencePieceProcessor(model_file="sp.model")
print(sp.encode("हिन्दी", out_type=str))
print(sp.encode("हिन्दी", out_type=int))
print(sp.encode("हिन्दी", out_type=str, enable_sampling=True, alpha=0.1, nbest_size=-1))Notes:
character_coverage=0.9995(the default) drops the rarest 0.05% of characters from the vocabulary. For a low-resource script, that can be most of its distinct characters. Set it to1.0if coverage matters more than vocabulary efficiency.byte_fallback=Trueis what produces<0xNN>tokens. It preserves round-trip but, as measured above, can dominate the output for scripts with no coverage.add_dummy_prefix=Trueinserts a leading space. It changes the first token and the offsets; remove it before aligning with the source.EncodeAsOffsetMappinggives character offsets;IsByte(id)identifies byte-fallback tokens.model_type="unigram"often segments agglutinative languages better than BPE because it optimizes a likelihood rather than greedily merging pairs.
Use it when you train from raw text and want a compact, portable model file, especially for multilingual corpora.
This is a loader, not a tokenizer implementation. It wraps a fast tokenizers
backend or a slow Python implementation.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B", use_fast=True)
print(tok.is_fast)
out = tok("हिन्दी", return_offsets_mapping=True, add_special_tokens=False)
print(out.tokens(), out["offset_mapping"])Notes:
return_offsets_mappingrequires a fast tokenizer. Checktok.is_fast.- The tokenizer can add special tokens by default; pass
add_special_tokens=Falsewhen measuring. tok.add_tokens([...])appends tokens but does not retrain merges; new tokens are matched greedily and can change segmentation of existing text.tok.train_new_from_iterator(...)trains a new tokenizer for the same model family, but the model's embeddings must then be resized.trust_remote_code=Trueexecutes code from the model repository. Do not use it on untrusted models.- Fast and slow implementations can differ on edge cases (normalization, whitespace). Pin the transformers version when reproducing counts.
Use it to load a published model's exact tokenizer. Do not use it as your
primary training API; tokenizers or sentencepiece is clearer.
- A SentencePiece model can be converted to a
tokenizerspipeline withconvert_slow_tokenizer, andtokenizerscan exporttokenizer.json. - A
tokenizersmodel can be saved astokenizer.jsonand loaded bytransformerswithuse_fast=True. - Byte-level BPE vocabularies from
tiktokenare not directly loadable bytokenizers; you can reconstruct the merge table, but special-token handling and the pre-tokenizer regex must be reproduced separately. - Changing a tokenizer changes token IDs. Any cached token sequence, prompt template, grammar constraint or fine-tuning dataset must be re-encoded.
Before adopting a tokenizer for a multilingual product:
- List the scripts and languages in scope, with expected corpus share.
- Measure fertility, bytes/token, raw-byte rate,
[UNK]rate and cluster-break rate on a per-language corpus, not on one sentence. - Check the vocabulary composition for each script (the table in section 3 is the shape of the check).
- Decide the normalization form and enforce it at one boundary.
- Decide how streaming and truncation handle grapheme clusters.
- Decide whether token-level metrics are comparable; if not, switch to bits-per-byte or per-character normalization.
- If the answer is "train a new tokenizer", budget for the embedding resize and continued pretraining, not just the tokenizer training.
- If the answer is "use a bigger vocabulary", verify on your languages; the Cherokee result above shows that size alone is not monotone.
Versions: tiktoken 0.14.0, tokenizers 0.23.3, transformers 5.22.1,
sentencepiece 0.2.3. Models: cl100k_base, o200k_base, gpt2,
xlm-roberta-base, bert-base-multilingual-cased, Mistral-7B-v0.1,
Qwen2.5-0.5B. Counts exclude BOS/EOS/CLS/SEP/PAD; [UNK] is excluded from
the content-token count and reported separately.
| sample | chars | bytes | words | cl100k | o200k | gpt2 | XLM-R | mBERT | Mistral | Qwen2.5 |
|---|---|---|---|---|---|---|---|---|---|---|
en |
44 | 44 | 9 | 10 | 10 | 10 | 13 | 13 | 12 | 10 |
fr |
40 | 44 | 5 | 13 | 12 | 17 | 12 | 11 | 14 | 13 |
de |
65 | 66 | 7 | 16 | 11 | 26 | 12 | 12 | 17 | 16 |
vi |
33 | 44 | 7 | 16 | 9 | 31 | 8 | 8 | 21 | 9 |
ru |
64 | 119 | 9 | 36 | 15 | 68 | 14 | 12 | 24 | 21 |
el |
39 | 71 | 7 | 34 | 12 | 43 | 8 | 12 | 39 | 31 |
he |
31 | 56 | 6 | 29 | 13 | 34 | 12 | 11 | 31 | 14 |
ar |
37 | 68 | 6 | 23 | 11 | 32 | 9 | 10 | 29 | 12 |
ar-harakat |
34 | 65 | 3 | 30 | 22 | 38 | 25 | 26 | 31 | 29 |
hi |
48 | 126 | 10 | 49 | 14 | 73 | 11 | 12 | 48 | 46 |
bn |
49 | 135 | 7 | 62 | 14 | 98 | 9 | 10 | 55 | 53 |
ta |
45 | 123 | 6 | 63 | 15 | 123 | 12 | 11 | 59 | 50 |
th |
34 | 102 | 1 | 36 | 13 | 68 | 9 | 21 | 34 | 17 |
am |
26 | 70 | 5 | 65 | 48 | 66 | 5 | 0 | 52 | 31 |
ka |
39 | 107 | 5 | 73 | 12 | 103 | 10 | 18 | 39 | 39 |
hy |
30 | 57 | 4 | 57 | 13 | 57 | 11 | 13 | 34 | 30 |
tr |
58 | 64 | 8 | 22 | 19 | 32 | 15 | 16 | 31 | 19 |
fi |
40 | 40 | 5 | 14 | 13 | 17 | 9 | 13 | 17 | 14 |
fi-word |
12 | 12 | 1 | 5 | 4 | 5 | 4 | 4 | 5 | 5 |
hu |
33 | 35 | 5 | 16 | 13 | 20 | 9 | 10 | 16 | 16 |
sw |
68 | 68 | 9 | 27 | 19 | 29 | 15 | 14 | 31 | 27 |
zh |
15 | 45 | 1 | 17 | 11 | 24 | 7 | 15 | 15 | 7 |
ja |
19 | 57 | 1 | 21 | 16 | 28 | 10 | 16 | 21 | 14 |
ko |
20 | 52 | 4 | 23 | 14 | 46 | 11 | 12 | 22 | 15 |
bo |
31 | 93 | 1 | 64 | 48 | 85 | 3 | 16 | 45 | 42 |
chr |
21 | 55 | 4 | 52 | 55 | 52 | 4 | 1 | 55 | 33 |
emoji-zwj |
7 | 25 | 1 | 18 | 11 | 14 | 7 | 0 | 19 | 10 |
emoji-skin |
2 | 8 | 1 | 6 | 3 | 5 | 2 | 0 | 2 | 2 |
emoji-flag |
4 | 14 | 1 | 9 | 6 | 9 | 4 | 0 | 10 | 5 |
combining |
2 | 3 | 1 | 2 | 2 | 3 | 1 | 2 | 2 | 1 |
precomposed |
1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
mBERT counts of 0 or 1 on am, chr, emoji-* mean the content was
[UNK]: Amharic 6 UNK, Cherokee 4 UNK, each emoji row 1 UNK. XLM-R also
emits 4 UNK on bo and 4 on chr, so its counts there are content tokens
only.
| sample | cl100k | o200k | gpt2 | XLM-R | mBERT | Mistral | Qwen2.5 |
|---|---|---|---|---|---|---|---|
en |
1.1 | 1.1 | 1.1 | 1.4 | 1.4 | 1.3 | 1.1 |
fr |
2.6 | 2.4 | 3.4 | 2.4 | 2.2 | 2.8 | 2.6 |
de |
2.3 | 1.6 | 3.7 | 1.7 | 1.7 | 2.4 | 2.3 |
vi |
2.3 | 1.3 | 4.4 | 1.1 | 1.1 | 3.0 | 1.3 |
ru |
4.0 | 1.7 | 7.6 | 1.6 | 1.3 | 2.7 | 2.3 |
el |
4.9 | 1.7 | 6.1 | 1.1 | 1.7 | 5.6 | 4.4 |
he |
4.8 | 2.2 | 5.7 | 2.0 | 1.8 | 5.2 | 2.3 |
ar |
3.8 | 1.8 | 5.3 | 1.5 | 1.7 | 4.8 | 2.0 |
ar-harakat |
10.0 | 7.3 | 12.7 | 8.3 | 8.7 | 10.3 | 9.7 |
hi |
4.9 | 1.4 | 7.3 | 1.1 | 1.2 | 4.8 | 4.6 |
bn |
8.9 | 2.0 | 14.0 | 1.3 | 1.4 | 7.9 | 7.6 |
ta |
10.5 | 2.5 | 20.5 | 2.0 | 1.8 | 9.8 | 8.3 |
am |
13.0 | 9.6 | 13.2 | 1.0 | 0.0 | 10.4 | 6.2 |
ka |
14.6 | 2.4 | 20.6 | 2.0 | 3.6 | 7.8 | 7.8 |
hy |
14.2 | 3.2 | 14.2 | 2.8 | 3.2 | 8.5 | 7.5 |
tr |
2.8 | 2.4 | 4.0 | 1.9 | 2.0 | 3.9 | 2.4 |
fi |
2.8 | 2.6 | 3.4 | 1.8 | 2.6 | 3.4 | 2.8 |
hu |
3.2 | 2.6 | 4.0 | 1.8 | 2.0 | 3.2 | 3.2 |
sw |
3.0 | 2.1 | 3.2 | 1.7 | 1.6 | 3.4 | 3.0 |
ko |
5.8 | 3.5 | 11.5 | 2.8 | 3.0 | 5.5 | 3.8 |
chr |
13.0 | 13.8 | 13.0 | 1.0 | 0.2 | 13.8 | 8.2 |
Percentages are raw-byte tokens / tokens and interior token boundaries not on a
UAX #29 cluster boundary / interior boundaries. n/a means the tokenizer
emitted [UNK] or its pieces do not reconstruct the source (WordPiece).
| sample | cl100k | o200k | XLM-R | Mistral | Qwen2.5 |
|---|---|---|---|---|---|
ru |
0% / 0% | 0% / 0% | 0% / 0% | 0% / 0% | 0% / 0% |
el |
18% / 9% | 0% / 0% | 0% / 0% | 0% / 0% | 0% / 0% |
he |
31% / 25% | 0% / 0% | 0% / 0% | 0% / 0% | 0% / 0% |
ar |
0% / 0% | 0% / 0% | 0% / 0% | 0% / 0% | 0% / 0% |
ar-harakat |
7% / 52% | 0% / 67% | 0% / 50% | 0% / 47% | 0% / 54% |
hi |
35% / 69% | 0% / 15% | 0% / 0% | 0% / 36% | 0% / 67% |
bn |
44% / 72% | 0% / 38% | 0% / 0% | 16% / 50% | 0% / 67% |
ta |
44% / 71% | 0% / 43% | 0% / 18% | 36% / 52% | 0% / 63% |
th |
11% / 29% | 0% / 8% | 0% / 0% | 0% / 21% | 0% / 38% |
am |
92% / 67% | 46% / 47% | 0% / n/a | 75% / 51% | 0% / 30% |
ka |
47% / 47% | 0% / 0% | 0% / 0% | 0% / 0% | 0% / 0% |
hy |
95% / 48% | 8% / 8% | 0% / 0% | 24% / 12% | 0% / 0% |
bo |
55% / 67% | 50% / 70% | 0% / n/a | 47% / 52% | 0% / 49% |
chr |
92% / 67% | 93% / 63% | 0% / n/a | 93% / 63% | 0% / 47% |
emoji-zwj |
61% / 100% | 36% / 100% | 0% / 100% | 84% / 100% | 0% / 100% |
Cluster-break rates for gpt2 are higher than cl100k on nearly every row;
they are omitted here for space, and visible in the earlier section where
relevant.
| id | text |
|---|---|
en |
The quick brown fox jumps over the lazy dog. |
fr |
L'élève étudie la linguistique comparée. |
de |
Die Sprachwissenschaft beschäftigt sich mit der Mehrsprachigkeit. |
vi |
Học sinh học tiếng Việt mỗi ngày. |
ru |
Москва — столица России, а Санкт-Петербург — культурная столица. |
el |
Η γλώσσα είναι το σπίτι του πολιτισμού. |
he |
השפה העברית היא שפה שמית עתיקה. |
ar |
اللغة العربية من أقدم اللغات السامية. |
ar-harakat |
اللُّغَةُ العَرَبِيَّةُ جَمِيلَةٌ. |
hi |
हिन्दी भारत की सबसे अधिक बोली जाने वाली भाषा है। |
bn |
বাংলা ভাষা দক্ষিণ এশিয়ার একটি গুরুত্বপূর্ণ ভাষা। |
ta |
தமிழ் மொழி மிகப் பழமையான மொழிகளில் ஒன்றாகும். |
th |
ภาษาไทยเป็นภาษาที่มีเสียงวรรณยุกต์ |
am |
አማርኛ የኢትዮጵያ የመንግስት ቋንቋ ነው። |
ka |
ქართული ენა ქართველების მშობლიური ენაა. |
hy |
Հայերենը հնդեվրոպական լեզու է։ |
tr |
Türkçe, eklemeli bir dil olduğu için sözcükler uzayabilir. |
fi |
Suomen kieli on agglutinatiivinen kieli. |
fi-word |
talossanikin |
hu |
A magyar nyelv agglutináló nyelv. |
sw |
Kweli Kiswahili ni lugha ya Kibantu inayozungumzwa Afrika Mashariki. |
zh |
中文是世界上最古老的语言之一。 |
ja |
日本語は文法が英語と大きく異なります。 |
ko |
한국어는 교착어로서 조사가 발달했다. |
bo |
བོད་ཡིག་ནི་བོད་ཀྱི་ཡིག་སྐད་ཡིན། |
chr |
ᏣᎳᎩ ᎦᏬᏂᎯᏍᏗ ᎠᏂᏴᏫ ᎤᏤᎵᎢ. |
emoji-zwj |
👨👩👧👦 |
emoji-skin |
👍🏽 |
emoji-flag |
🏳️🌈 |
combining |
e + U+0301 (NFD) |
precomposed |
é (NFC) |
The following is the essential part of the harness used for the tables. It
requires tiktoken, transformers, tokenizers, sentencepiece and regex.
import json, regex, tiktoken
from transformers import AutoTokenizer
def gpt2_bytes_to_unicode():
bs = list(range(33, 127)) + list(range(161, 173)) + list(range(174, 256))
cs = bs[:]; n = 0
for b in range(256):
if b not in bs:
bs.append(b); cs.append(256 + n); n += 1
return dict(zip(bs, (chr(c) for c in cs)))
U2B = {v: k for k, v in gpt2_bytes_to_unicode().items()}
class Tik:
family = "byte"
def __init__(self, name):
self.enc = tiktoken.get_encoding(name)
self.special = set(self.enc._special_tokens.values())
def encode(self, s):
return self.enc.encode(s)
def piece_bytes(self, i):
return self.enc.decode_single_token_bytes(i)
def piece(self, i):
return self.piece_bytes(i).decode("utf-8", "backslashreplace")
def is_special(self, i):
return i in self.special
class HF:
def __init__(self, name):
self.t = AutoTokenizer.from_pretrained(name)
joined = "".join(self.t.tokenize("hello world"))
self.family = ("sp" if "\u2581" in joined
else "byte" if "\u0120" in joined else "wordpiece")
self.special = set(self.t.all_special_ids)
def encode(self, s):
return self.t.encode(s)
def piece(self, i):
p = self.t.convert_ids_to_tokens(i)
return p[0] if isinstance(p, list) else p
def piece_bytes(self, i):
p = self.piece(i)
if p.startswith("<0x") and p.endswith(">") and len(p) == 6:
return bytes([int(p[3:5], 16)])
if self.family == "sp":
return p.replace("\u2581", " ").encode("utf-8")
if self.family == "byte":
return bytes(U2B.get(c, ord(c) & 0xFF) for c in p)
return (p[2:] if p.startswith("##") else p).encode("utf-8")
def is_special(self, i):
return i in self.special
def measure(model, text):
structural = {"<s>", "</s>", "<pad>", "[CLS]", "[SEP]", "[PAD]", "<mask>"}
ids, unk = [], 0
for i in model.encode(text):
p = model.piece(i)
if p in structural: continue
if p in ("<unk>", "[UNK]"): unk += 1; continue
ids.append((i, p))
raws = [model.piece_bytes(i) for i, _ in ids]
# Remove the dummy prefix space that SentencePiece inserts.
if raws and not text.startswith(" ") and raws[0].startswith(b" "):
raws[0] = raws[0][1:]
if not raws[0]:
raws.pop(0); ids.pop(0)
offs, breaks, fallback = [0], 0, 0
clusters, pos, starts = regex.findall(r"\X", text), 0, set()
for c in clusters:
starts.add(pos); pos += len(c.encode())
starts.add(pos)
for i, b in zip([i for i, _ in ids], raws):
p = model.piece(i)
fallback += (bool(regex.fullmatch(r"<0x[0-9A-F]{2}>", p))
if isinstance(model, HF)
else len(b) == 1 and b[0] >= 0x80)
offs.append(offs[-1] + len(b))
breaks = sum(1 for o in offs[1:-1] if o not in starts)
return {"tokens": len(ids), "unk": unk, "fallback": fallback,
"breaks": breaks, "bytes": len(text.encode())}Caveats, repeated because they matter:
- The sample set is 31 short sentences, chosen to exercise mechanics. It is not a corpus.
- Token counts exclude structural specials.
[UNK]is counted separately and excluded fromtokens. - Cluster-break rates are reported only for tokenizers whose pieces reconstruct the source bytes exactly. WordPiece's pre-tokenizer drops whitespace around punctuation, so mBERT is excluded from that column.
- The vocabulary classification in section 3 is a heuristic: majority script of alphabetic characters per entry, with non-letter and fragment entries unclassified. Treat the percentages as lower bounds.
- P. Sennrich, B. Haddow, A. Birch. "Neural Machine Translation of Rare Words with Subword Units." ACL 2016. arXiv:1508.07909.
- T. Kudo, J. Richardson. "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing." EMNLP 2018 (demo). arXiv:1808.06226.
- A. Petrov, E. La Malfa, P. Torr, A. Bibi. "Language Model Tokenizers Introduce Unfairness Between Languages." NeurIPS 2023. arXiv:2305.15425.
- L. Xue et al. "ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models." TACL 2022. arXiv:2105.13626.
- Unicode Standard Annex #29, "Unicode Text Segmentation."
- Unicode Standard Annex #15, "Unicode Normalization Forms."
- RFC 3629, "UTF-8, a transformation format of ISO 10646."
- OpenAI
tiktoken: https://github.com/openai/tiktoken - Hugging Face
tokenizers: https://huggingface.co/docs/tokenizers - Google
sentencepiece: https://github.com/google/sentencepiece - GPT-2
encoder.py: https://github.com/openai/gpt-2