KorByte-128K

KorByte-128K is a Korean-focused, Unicode-aware byte-level BPE tokenizer with 128,000 learned tokens and 256 stable special-token IDs. It performs no Unicode normalization, so it preserves spaces, line endings, decomposed Hangul, emoji, and arbitrary UTF-8 text exactly.

It ranks first among 8 successfully loaded, revision-pinned public systems by both fertility and effective bits per byte (EBPB) on the Korean slice of the Multilingual Tokenizer Benchmark. It also ranks first on a frozen, post-selection KMMLU test audit. This is a scoped intrinsic result, not proof of universal or downstream language-model superiority.

Quick start

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("dawncr0w/KorByte-128K", use_fast=True)
text = "새 기능을 배포하기 전에 테스트 결과를 확인해 주세요."
ids = tokenizer.encode(text, add_special_tokens=False)
assert tokenizer.decode(ids, clean_up_tokenization_spaces=False) == text

Measured result

• Public Korean benchmark fertility / EBPB: 1.4463 / 3.0710 (rank 1)
• KMMLU audit fertility / EBPB: 1.9364 / 3.2942 (rank 1)
• Macro token reduction vs. kakaocorp/kanana-2-3b-base: 18.92%
• Public first-place gate: PASS
• Exact round-trip release validation: passed
• Core / total vocabulary: 128,000 / 128,256

See reports/comparison.md for the pinned public ranking and unavailable artifacts, and reports/research.md for the accepted and rejected variants. KLUE domain counts and throughput are in reports/benchmark.md. Machine-readable evidence is under reports/.

Why OKT and MeCab-ko are not the primary baseline

OKT and MeCab-ko are morphological analyzers. They do not provide the same fixed-vocabulary, lossless, byte-complete encoding contract required by an LLM tokenizer. Their output counts and speed are reported as useful context; Kanana-2 is the like-for-like tokenizer baseline.

Design

• Unicode-aware word-boundary segmentation with six-digit number chunks
• Byte-level alphabet, decoder, and no normalizer for complete coverage
• 700 million-character Korean-heavy public training mixture with a smaller English allocation
• Deterministic source revisions, shuffle seed, filtering, deduplication, and manifests
• 256 contiguous special-token IDs from 128,000 through 128,255

Intended use and limitations

This artifact is intended for Korean-heavy language-model experiments, token-count analysis, and as a starting vocabulary for training a new model. Replacing the tokenizer of an existing model without retraining or vocabulary adaptation will break that model. Compression alone does not guarantee better accuracy, latency, safety, or training efficiency. The Thunder public artifact could not be loaded because it uses a custom tokenizer model; the comparison report records the exact failure instead of silently omitting it.

Reproduce

uv sync --all-extras
uv run korbyte prepare --scale 1.0
uv run korbyte train
uv run korbyte benchmark
uv run korbyte compare
uv run korbyte render
uv run korbyte validate

The prepared training text is intentionally excluded from this repository. Exact source revisions, accepted character counts, filtering, and hashes are documented in DATA_SOURCES.md and provenance/.