PRAIG/smb-staff-abc-bpe

SMB staff-synth ABC BPE tokenizer

BPE tokenizer trained on the ABC labels of the SMB staff-synth dataset (218 train samples), for use as a source in METI.

• Model: BPE, vocab_size=1000, min_frequency=2
• Pre-tokenizer/decoder: Metaspace (whitespace split, exact decode)
• Special tokens: <pad>, <blank>, <bos>, <eos>, <unk>
• No static pretokenization: BPE runs over raw ABC text

Usage in METI

tokenizer:
  _target_: meti.data.tokenizers.BPETokenizer
  _partial_: true
  source: <user>/<repo>

Standalone

from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download

tok = Tokenizer.from_file(hf_hub_download("<user>/<repo>", "tokenizer.json"))