oghuz-tokenizer

A 65,536-piece byte-level BPE tokenizer built from scratch for the Oghuz language family: Turkish first, with Azerbaijani and Turkmen. The first public artifact of the Oghuz project, a fully open, fully documented pan-Oghuz language model effort by Emre Yavuz. Every fertility number below was measured on a frozen, hash-committed measurement set that ships in this repository, so you can re-verify rather than trust.

Status: v1.0.0, 2026-07-12. No released model uses this tokenizer yet; the Oghuz models will, and the vocabulary is frozen across that whole model ladder (the 60 reserved special slots exist so future control tokens never force retokenization). Tested with transformers 5.13.1 / tokenizers 0.22.2 (full environment in manifests/environment.txt); any transformers version with PreTrainedTokenizerFast + chat-template support works.

Results, stated plainly

We targeted under 1.6 tokens per Turkish word. We measured 1.762. The target was committed in the project's decision register before this measurement set existed, derived from fertility numbers reported in the literature on each project's own test sets. Our measurement set is harder on purpose: nine register files (five Turkish registers: news, encyclopedic, literature, legal, conversational; plus encyclopedic and conversational for Azerbaijani and Turkmen), frozen and hash-committed BEFORE any Oghuz tokenizer was trained, and the vocabulary deliberately spends part of its budget on a first-class code, English, and math reserve. The target is not retro-revised. The full evidence chain (git history binding the set hash and the target to dates before training) becomes publicly checkable when the project repository is published with the first model release; until then, everything in this repository is internally verifiable: python verify.py recomputes the set hashes and the headline number.

Measured baselines (fertility: tokens per word, lower is better; word-weighted pooled)

Tokenizer	vocab	Turkish	Azerbaijani	Turkmen
oghuz-tokenizer (this)	65,536	1.762	2.291	2.585
Kumru (Turkish specialist)	50,176	1.874	3.612	3.607
Llama-3	128,256	2.409	3.680	3.300
o200k (GPT-4o class)	200,019	2.435	2.808	2.679
Gemma-3	262,145	2.471	3.339	3.280
Qwen3	151,669	2.793	3.879	3.459

Reading: among the tokenizers we measured, the frontier multilingual ones spend 37 to 59 percent more tokens per Turkish word than this one; the best Turkish-specialist fertility we measured (Kumru) is beaten by 6 percent, in every register, by a tokenizer that also carries code, English, and math. To our knowledge no publicly released tokenizer specializes in Azerbaijani or Turkmen; this one is trained on all three languages. Fairness caveats: the multilingual baselines carry 100+ languages, which is part of why their Turkish fertility is high and their AZ/TK columns are what they are; Llama-3 and Gemma-3 tokenizer files came from revision-pinned community mirrors (official repos are gated), recorded in manifests/baselines.json. Fewer tokens per word means lower cost, lower latency, and more effective context for the same window, in every model that uses this tokenizer.

What the losing arm was better at (reported, not hidden)

A SentencePiece Unigram arm was trained on identical data under a pre-committed decision rule. Vocabulary-level diagnostics (morphological analyzer over the vocab, not the measurement set): the Unigram vocabulary contained more valid standalone Turkish words (66.6 percent of alphabetic pieces vs 57.7) and aligned better with unsupervised morpheme boundaries (Morfessor F1 0.457 vs 0.308), and it cost 9.38 percent more tokens per word. The rule, locked before any number existed, picks fertility; the trade-off is documented here rather than traded away silently.

Design

• 65,536 total pieces (2^16), byte fallback via the full byte-level alphabet: no unknown tokens, ever, in any language or script.
• NFC normalization is baked into the tokenizer itself; Turkish-aware casing rules were enforced across the entire build pipeline (never locale-blind lowercasing).
• Modern GPT-4-style split pattern whose optional-punctuation prefix naturally isolates Turkish apostrophe suffixes (İstanbul'da becomes İstanbul + 'da, so the vocabulary learns 'da, 'nin and friends as pieces; verified in the shipped vocab), single-digit number segmentation, and full-document training so code indentation and newline pieces are learned.
• ChatML special tokens plus 60 reserved slots.

Special tokens

id	token	role
0	<|endoftext|>	eos; no post-processor is configured, so append it yourself when packing training data
1	<|im_start|>	ChatML turn open
2	<|im_end|>	ChatML turn close
3	<|pad|>	padding
4-63	<|reserved_00|> .. <|reserved_59|>	reserved for future control tokens

model_max_length is intentionally unset (set it from your model config); bos_token is null.

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("yavuz-ai/oghuz-tokenizer")
tok.encode("Kıyıda üç kırmızı balık yüzüyordu.")

# Chat (ChatML): control tokens land as real special ids (1, 2)
ids = tok.apply_chat_template(
    [{"role": "user", "content": "selam"}],
    tokenize=True, add_generation_prompt=True,
)

Encoding untrusted text: special-token strings are matched by default (standard transformers behavior, required for chat templates to work). If you encode raw user input OUTSIDE a chat template and want <|im_end|> typed by a user to stay plain text, pass split_special_tokens=True for that call:

tok("user supplied text", split_special_tokens=True)

Verify our numbers yourself

python verify.py (needs only transformers) recomputes the per-file SHA-256 hashes, the set hash, and the pooled Turkish fertility, and compares them to the published values. The set hash is SHA-256 over the nine register files concatenated in sorted filename order (cat measurement-set/*.txt | shasum -a 256 gives dc9733aea2ebf9bcf92629a8ee55e292dd33694ef3cf9fdbecea217f33fb8524). The measurement protocol ships as MEASUREMENT-PROTOCOL.md; the word definition is stated operationally (Python str.split() and str.isalpha()) so independent implementations agree exactly. Per-source revisions, licenses, and hashes: manifests/. Note: verification of the published numbers uses the shipped frozen text; regenerating the set from its sources additionally needs the archived dump and export files kept in the project repository (Wikimedia rotates its dumps).

Licensing

Apache-2.0 covers the tokenizer (tokenizer.json), configs, recipes, manifests, and verify.py. The measurement-set/ texts are verbatim third-party excerpts and carry their sources' licenses (Wikipedia CC BY-SA 4.0, Wikinews CC BY 2.5, Tatoeba CC BY 2.0 FR, Turkish statutes public domain under FSEK Art. 31): see measurement-set/DATA-LICENSES.md for per-file licenses, attribution, and the modification notice. All register text is unmodified source excerpt; any opinions or characterizations in it belong to the sources.

Provenance and disclosure

Training sample: 3.0GB across Turkish, Azerbaijani, and Turkmen web text (FineWeb-2, ODC-By), English (FineWeb-Edu), code (The Stack sample; predominantly permissive with some weak-copyleft MPL/EPL records, per-record license tally in the manifest; the code is used for training only, not redistributed), and math, in the documented blend, decontaminated against the measurement set with removals recorded in the manifest. Trained on a single consumer machine; no cloud compute. Engineering was AI-assisted throughout (Claude); design decisions, quality gates, and this release were human-reviewed and human-approved.

Citation

@misc{oghuz_tokenizer_2026,
  title  = {Oghuz Tokenizer: a 64k byte-level BPE for Turkish, Azerbaijani, and Turkmen},
  author = {Yavuz, Emre},
  year   = {2026},
  url    = {https://huggingface.co/yavuz-ai/oghuz-tokenizer}
}

Contact

Emre Yavuz · https://yavuz.ai · https://huggingface.co/yavuz-ai · https://x.com/yavuzai