README.md

---
license: apache-2.0
language:
  - multilingual
tags:
  - tokenizer
  - bpe
  - byte-level-bpe
  - chatml
  - routing
  - moe
---

Enjoy — We extend the JiRack Models Ecosystem! 🚀
- The Best RAG with Mixture-of-Experts via CMS Manhattan RAG System with JiRack Ternary Experts for CPU Solution.
- JiRackTernary_1b model https://huggingface.co/kgrabko/JiRackTernary_1b

# JiRack Router Tokenizer Pro

**JiRack Pro Tokenizer - 347 active language editions of Wikipedia** 

**High-performance custom Byte-Level BPE tokenizer** trained on the full Wikipedia dump across multiple languages.

Developed specifically for intelligent routing models and Mixture-of-Experts systems.


## Model Details

- **Developer**: Konstantin Grabko (JiRack)
- **License**: Apache License 2.0
- **Training Data**: Full Wikipedia multilingual dump
- **Vocabulary Size**: 65,536 tokens
- **Special Tokens**: 128 reserved tokens (including 40+ domain routing tokens)

## Key Features

- Correctly placed `<|unk|>` token at ID 0
- Full native support for **ChatML** format (`<|im_start|>` / `<|im_end|>`)
- Large set of specialized routing tokens (`__CODING__`, `__MATH__`, `__PYTHON__`, `__SCIENCE__`, etc.)
- Support JiRack Robotics Technology with tags  (`<|action_start|>` / `<|action_end|>`)
- Support JiRack vision , images , audio-visual
- Strong multilingual performance

# Basic system and dialogue tokens
"<|unk|>",
"<|endoftext|>",
"<|padding|>",
"<|im_start|>",
"<|im_end|>",
# Core roles
"<|im_start|>system",
"<|im_start|>user",
"<|im_start|>assistant",
# Additional useful tokens
"<|im_start|>tool",
"<|im_start|>function",
# Reasoning block
"<|im_start|>thought",
# Tool calls
"<|tool_call|>",
"<|tool_response|>",
# Multimodality and audio-visual block
"<|image|>",
"<|video|>",
"<|sound|>",
"<|voice|>",
"<|listening|>",
"<|vision|>",
# Emotional state (Mood)
"<|mood_happy|>",
"<|mood_sad|>",
"<|mood_angry|>",
"<|mood_neutral|>",
# --- FIM (Fill-in-the-Middle) tokens from StarCoder ---
"<|fim_prefix|>",
"<|fim_middle|>",
"<|fim_suffix|>",
# Robotics (trajectory/command boundaries)
"<|action_start|>",
"<|action_end|>",


# Highlights
- **Architecture**: Byte-Level BPE (Byte-level Byte Pair Encoding) which natively prevents Out-Of-Vocabulary (OOV) tokens.
- **Form Factor**: Fully wrapped into Hugging Face `PreTrainedTokenizerFast` with native `ByteLevel` decoders for clean cyrillic representation.
- **Chat Standard**: Out-of-the-box support for **ChatML** formatting (`<|im_start|>` / `<|im_end|>`).
- **Domain Specialization**: Pre-baked atomic routing tokens like `__CODING__` and `__PYTHON__` etc.

### CMS Manhattan Service & Support 

- Jirack patent guards your technology for competitors 
- Redesign Llama , Qwen , Gemma to Ternary model
- Re-tain and replace embeddings for  Llama , Qwen , Gemma  to extend langeages to 347
- Accelerate inference via high compression 256K tokenizer and replace multiplication with sum operations via Ternary weights

## Usage

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("CMSManhattan/JiRack-Router-Tokenizer-65K")

text = "<|im_start|>user\n__CODING__ __PYTHON__ Write a merge sort function.<|im_end|>"
print(tokenizer.encode(text))
```

### JiRack Pretrain Dataset
https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset

```bash
python train_jirack_accelerate.py
Processing jirack_pretrain_chunk_0.pt:  27%|█████████████████████▋                                                  | 268/1000 [22:02:45<60:12:33, 296.11s/it, loss=6.3145, avg_loss=7.0132, ppl=1111.16]
Processing jirack_pretrain_chunk_0.pt:  87%|███████████████████████████████████████████████████████████████████████      | 866/1000 [75:57:47<13:13:29, 355.29s/it, loss=2.8616, avg_loss=5.9877, ppl=398.52]
```

### Benchmark for tokens  quality .


```bash
=== Text after ChatML Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Write a merge sort function in Python.<|im_end|>


=== Tokens (IDs) ===
[5, 326, 5095, 944, 396, 23348, 1021, 7831, 5869, 141, 4, 326, 6, 326, 29, 348, 44, 26876, 396, 52698, 6521, 2031, 460, 7524, 141, 4, 326]

=== Decoding Token by Token ===
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
5 -> '<|im_start|>system'
326 -> '
'
5095 -> 'You'
944 -> ' are'
396 -> ' a'
23348 -> ' precise'
1021 -> ' ro'
7831 -> 'uter'
5869 -> ' model'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
6 -> '<|im_start|>user'
326 -> '
'
29 -> '__CODING__'
348 -> ' '
44 -> '__PYTHON__'
26876 -> ' Write'
396 -> ' a'
52698 -> ' merge'
6521 -> ' sort'
2031 -> ' function'
460 -> ' in'
7524 -> ' Python'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
```

## 📧 Contact & Licensing

For joint ventures, hardware integration, or licensing inquiries:

- **Email:** grabko@cmsmanhattan.com
- **Phone:** +1 (516) 777-0945
- **Location:** New York, USA

## 📧 Copyright 2026  CMS Manhattan . All rights reserved