📌 Overview
EspBPE-49K is a custom, hyper-optimized 49,152-vocabulary Byte-Level Byte Pair Encoding (BPE) tokenizer built specifically for modern Spanish Large Language Models (LLMs).
Trained across a massive multi-domain Spanish dataset mixture, EspBPE-49K achieves an extraordinary compression ratio (~4.8 to 5.1 characters per token), preserving contextual length and embedding space efficiency across formal encyclopedic text, technical documentation, sports commentary, and regional internet conversations.
⚡ Quickstart
You can load EspBPE-49K directly using the Hugging Face transformers library:
from transformers import AutoTokenizer
# Load EspBPE-49K directly from Hugging Face Hub
tokenizer = AutoTokenizer.from_pretrained("DinoResearch/EspBPE-49K")
# Sample text
text = "Estamos entrenando un tokenizer en español super rápido y eficiente con 49,152 de vocabulario! 🚀"
# Encode text into token IDs
input_ids = tokenizer.encode(text)
tokens = tokenizer.convert_ids_to_tokens(input_ids)
print(f"Total Tokens: {len(input_ids)}")
print("Tokens:", tokens)
print("Decoded:", tokenizer.decode(input_ids))
📊 Performance & Compression Ratio
| Text Type | Character Count | Token Count | Compression Ratio |
|---|---|---|---|
| Sports Commentary | 142 chars | 28 tokens | ~5.07 chars / token |
| Encyclopedic History (Wikipedia) | 245 chars | 49 tokens | ~5.00 chars / token |
| General Spanish Sentence | 107 chars | 22 tokens | ~4.86 chars / token |
🗃️ Training Dataset Mixture
EspBPE-49K was trained on a carefully balanced multi-source Spanish corpus:
- FineWeb-2 (
spa_Latn): Modern, high-quality web crawl text. - Spanish Wikipedia (
20231101.es): Encyclopedic knowledge baseline. - CulturaX (
es): Filtered, clean multilingual web data (OSCAR + mC4). - Spanish Tweets (
pysentimiento): Informal dialogue, regional accents, and modern social media slang. - PleIAs Spanish PD-Books: Classic literature and historical Spanish prose.
🌟 Key Highlights & Features
- Single-Token Complex Vocabulary: Long formal words like
inconstitucionalidad,extraordinariamente,descentralización,afortunadamente, andRehabilitaciónmap directly to single Token IDs. - Native Accent & UTF-8 Support: Seamless byte-level decoding for accented characters (
á,é,í,ó,ú,ñ) without producing<unk>errors. - Native Emoji Primitives: High-frequency multi-byte emojis (e.g.,
🤪,😌,🇵🇷,🟢) are learned as native single-token lookup entries. - Regional & Technical Fluency: Optimized subword roots for tech terms (
javascript,tensor,Unix,coaxial) alongside regional slang across Spanish-speaking countries.
📜 Model Card Details
- Vocabulary Size: 49,152 (
49k) - Model Type: ByteLevel BPE
- Space Prefix:
Ġ(ByteLevel space representation) - Organization: DinoResearch## ⚔️ Benchmark: EspBPE-49K vs. OpenAI o200k_base (GPT-4o)
To evaluate real-world web performance, EspBPE-49K was benchmarked head-to-head against OpenAI's o200k_base tokenizer (used in GPT-4o) across 500 unfiltered Spanish web documents (2,292,109 raw characters) from the FineWeb-2 (spa_Latn) dataset stream.
🏆 500-Text FineWeb-2 Benchmark Results
| Metric | 🦖 EspBPE-49K | 🌐 o200k_base (GPT-4o) | Advantage / Delta |
|---|---|---|---|
| Vocabulary Size | 49,152 | 200,019 | ~4x Smaller Table |
| Total Encoded Tokens | 484,581 | 520,509 | -35,928 Tokens (-6.90%) |
| Compression Ratio | 4.730 chars/token | 4.404 chars/token | +0.326 chars/token |
| Document Match Wins | 441 / 500 (88.2%) 🥇 | 50 / 500 (10.0%) | Landslide Victory |
| Draws / Ties | 9 / 500 (1.8%) | 9 / 500 (1.8%) | — |
🧠 Benchmark Key Takeaways
- Massive Parameter Savings: EspBPE-49K achieves superior compression using less than 1/4th of
o200k_base's vocabulary size, saving tens of millions of embedding parameters during pretraining. - 6.90% Token Reduction: Encodes identical Spanish text into 6.90% fewer overall tokens, directly reducing memory requirements and accelerating autoregressive pretraining speed.
- Extended Context Window: Higher token efficiency means more raw text fits into a fixed context length (e.g., 2048 / 4096 tokens).