tcr-vtoken tokenizer

A residue-level TCR tokenizer where each V gene is one atomic token. Vocab = 26 base tokens (20 amino acids + specials) + 101 tidytcells-standardised V-gene tokens ([V:TRBV20-1], [V:TRAV1-2], …) + [V_UNK] = 128 total. It is built for the input grammar [V:gene] | CDR3, where the atomic V token stands in for the germline CDR1/CDR2 (which the V gene determines).

Files

Standard HF tokenizer (tokenizer.json, vocab.txt, added_tokens.json, …) + vgene_map.json — a map from a raw V-gene symbol to its atomic token ("TRBV20-1" -> "[V:TRBV20-1]"; unknown → [V_UNK]).

Use it

import json
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("argentel/tcr-vtoken")                 # base AA + V-gene tokens (128)
vmap = json.load(open(hf_hub_download("argentel/tcr-vtoken", "vgene_map.json")))   # raw V-gene -> token

v_gene, cdr3 = "TRBV20-1", "CASSYSTDTQYF"
vtok = vmap.get(v_gene, "[V_UNK]")                                         # atomic V token
text = f"{vtok} {tok.sep_token} {' '.join(cdr3)}"                          # "[V:TRBV20-1] | C A S S ..."
enc = tok(text)
print(tok.convert_ids_to_tokens(enc["input_ids"]))
# ['*', '[V:TRBV20-1]', '|', 'C', 'A', 'S', 'S', 'Y', 'S', 'T', 'D', 'T', 'Q', 'Y', 'F', '|']
#   *=[CLS] and |=[SEP] are added automatically; [V:TRBV20-1] is ONE atomic token.
print(tok.convert_tokens_to_ids("[V:TRBV20-1]"))                          # 87  (V tokens have ids >= 26)

The V-gene tokens live in added_tokens.json (ids 26–127), not vocab.txt (which is only the 26 base tokens); AutoTokenizer merges both, so len(tok) == 128.