| --- |
| license: apache-2.0 |
| language: |
| - multilingual |
| tags: |
| - tokenizer |
| - bpe |
| - byte-level-bpe |
| - chatml |
| - routing |
| - moe |
| --- |
| |
| Enjoy — We extend the JiRack Models Ecosystem! 🚀 |
| - The Best RAG with Mixture-of-Experts via CMS Manhattan RAG System with JiRack Ternary Experts for CPU Solution. |
| - JiRackTernary_1b model https://huggingface.co/kgrabko/JiRackTernary_1b |
|
|
| # JiRack Router Tokenizer Pro |
|
|
| **JiRack Pro Tokenizer - 347 active language editions of Wikipedia** |
|
|
| **High-performance custom Byte-Level BPE tokenizer** trained on the full Wikipedia dump across multiple languages. |
|
|
| Developed specifically for intelligent routing models and Mixture-of-Experts systems. |
|
|
|
|
| ## Model Details |
|
|
| - **Developer**: Konstantin Grabko (JiRack) |
| - **License**: Apache License 2.0 |
| - **Training Data**: Full Wikipedia multilingual dump |
| - **Vocabulary Size**: 65,536 tokens |
| - **Special Tokens**: 128 reserved tokens (including 40+ domain routing tokens) |
|
|
| ## Key Features |
|
|
| - Correctly placed `<|unk|>` token at ID 0 |
| - Full native support for **ChatML** format (`<|im_start|>` / `<|im_end|>`) |
| - Large set of specialized routing tokens (`__CODING__`, `__MATH__`, `__PYTHON__`, `__SCIENCE__`, etc.) |
| - Support JiRack Robotics Technology with tags (`<|action_start|>` / `<|action_end|>`) |
| - Support JiRack vision , images , audio-visual |
| - Strong multilingual performance |
|
|
| # Basic system and dialogue tokens |
| "<|unk|>", |
| "<|endoftext|>", |
| "<|padding|>", |
| "<|im_start|>", |
| "<|im_end|>", |
| # Core roles |
| "<|im_start|>system", |
| "<|im_start|>user", |
| "<|im_start|>assistant", |
| # Additional useful tokens |
| "<|im_start|>tool", |
| "<|im_start|>function", |
| # Reasoning block |
| "<|im_start|>thought", |
| # Tool calls |
| "<|tool_call|>", |
| "<|tool_response|>", |
| # Multimodality and audio-visual block |
| "<|image|>", |
| "<|video|>", |
| "<|sound|>", |
| "<|voice|>", |
| "<|listening|>", |
| "<|vision|>", |
| # Emotional state (Mood) |
| "<|mood_happy|>", |
| "<|mood_sad|>", |
| "<|mood_angry|>", |
| "<|mood_neutral|>", |
| # --- FIM (Fill-in-the-Middle) tokens from StarCoder --- |
| "<|fim_prefix|>", |
| "<|fim_middle|>", |
| "<|fim_suffix|>", |
| # Robotics (trajectory/command boundaries) |
| "<|action_start|>", |
| "<|action_end|>", |
| |
| |
| # Highlights |
| - **Architecture**: Byte-Level BPE (Byte-level Byte Pair Encoding) which natively prevents Out-Of-Vocabulary (OOV) tokens. |
| - **Form Factor**: Fully wrapped into Hugging Face `PreTrainedTokenizerFast` with native `ByteLevel` decoders for clean cyrillic representation. |
| - **Chat Standard**: Out-of-the-box support for **ChatML** formatting (`<|im_start|>` / `<|im_end|>`). |
| - **Domain Specialization**: Pre-baked atomic routing tokens like `__CODING__` and `__PYTHON__` etc. |
| |
| ### CMS Manhattan Service & Support |
| |
| - Jirack patent guards your technology for competitors |
| - Redesign Llama , Qwen , Gemma to Ternary model |
| - Re-tain and replace embeddings for Llama , Qwen , Gemma to extend langeages to 347 |
| - Accelerate inference via high compression 256K tokenizer and replace multiplication with sum operations via Ternary weights |
| |
| ## Usage |
| |
| ```python |
| from transformers import AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("CMSManhattan/JiRack-Router-Tokenizer-65K") |
|
|
| text = "<|im_start|>user\n__CODING__ __PYTHON__ Write a merge sort function.<|im_end|>" |
| print(tokenizer.encode(text)) |
| ``` |
| |
| ### JiRack Pretrain Dataset |
| https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset |
| |
| ```bash |
| python train_jirack_accelerate.py |
| Processing jirack_pretrain_chunk_0.pt: 27%|█████████████████████▋ | 268/1000 [22:02:45<60:12:33, 296.11s/it, loss=6.3145, avg_loss=7.0132, ppl=1111.16] |
| Processing jirack_pretrain_chunk_0.pt: 87%|███████████████████████████████████████████████████████████████████████ | 866/1000 [75:57:47<13:13:29, 355.29s/it, loss=2.8616, avg_loss=5.9877, ppl=398.52] |
| ``` |
| |
| ### Benchmark for tokens quality . |
| |
| |
| ```bash |
| === Text after ChatML Template === |
| <|im_start|>system |
| You are a precise router model.<|im_end|> |
| <|im_start|>user |
| __CODING__ __PYTHON__ Write a merge sort function in Python.<|im_end|> |
|
|
|
|
| === Tokens (IDs) === |
| [5, 326, 5095, 944, 396, 23348, 1021, 7831, 5869, 141, 4, 326, 6, 326, 29, 348, 44, 26876, 396, 52698, 6521, 2031, 460, 7524, 141, 4, 326] |
|
|
| === Decoding Token by Token === |
| [transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway. |
| 5 -> '<|im_start|>system' |
| 326 -> ' |
| ' |
| 5095 -> 'You' |
| 944 -> ' are' |
| 396 -> ' a' |
| 23348 -> ' precise' |
| 1021 -> ' ro' |
| 7831 -> 'uter' |
| 5869 -> ' model' |
| 141 -> '.' |
| 4 -> '<|im_end|>' |
| 326 -> ' |
| ' |
| 6 -> '<|im_start|>user' |
| 326 -> ' |
| ' |
| 29 -> '__CODING__' |
| 348 -> ' ' |
| 44 -> '__PYTHON__' |
| 26876 -> ' Write' |
| 396 -> ' a' |
| 52698 -> ' merge' |
| 6521 -> ' sort' |
| 2031 -> ' function' |
| 460 -> ' in' |
| 7524 -> ' Python' |
| 141 -> '.' |
| 4 -> '<|im_end|>' |
| 326 -> ' |
| ' |
| ``` |
| |
| ## 📧 Contact & Licensing |
| |
| For joint ventures, hardware integration, or licensing inquiries: |
| |
| - **Email:** grabko@cmsmanhattan.com |
| - **Phone:** +1 (516) 777-0945 |
| - **Location:** New York, USA |
| |
| ## 📧 Copyright 2026 CMS Manhattan . All rights reserved |
| |
