potion-multilingual-128M-i8-tokenade

An int8 quantisation of minishlab/potion-multilingual-128M, plus precomputed tokenizer artifacts for low-memory loading.

All credit for the model itself goes to the Minish Lab — this repo adds no training, no distillation and no new data. It only changes how the same weights are stored and loaded.

Why

model2vec inference is tokenize → look up rows → mean-pool: no context, no forward pass. Loading it should not cost like a transformer, but in practice a stock load decodes the whole weight matrix to f32 and inflates tokenizer.json into an in-memory trie — together about 1 GB of RSS and ~930 ms before the first vector. That is fine for a long-lived server and prohibitive for a short-lived CLI process.

These artifacts are the same model arranged so it can be memory-mapped instead:

	stock load	these artifacts
RSS	~1056 MB	~4.5 MB
first embedding	~930 ms	~1 ms
per embedding	~0.03 ms	~0.11 ms

The per-embedding cost regresses (no pre-decoded matrix, no cache); a process needs roughly 12 700 embeddings in one run before that outweighs the load-time saving.

Files

file	what it is
model.safetensors	the embedding matrix, I8, [500353, 256]
vocab.fst	the Unigram vocab as an FST: token bytes → id
vocab.fst.scores	f64 unigram log-probs, one per id, in id order
vocab.fst.norm	the normalizer / pre-tokenizer spec and decoding metadata
tokenizer.json	upstream's, unchanged — kept so the FST can be re-derived and verified
config.json	upstream's, unchanged

Quantisation

Symmetric, one global scale (max|W| / 127). The scale is deliberately not stored: mean-pooling is linear and model2vec L2-normalises its output, so a single global factor cancels exactly. Rows can therefore be read straight from disk as raw i8.

This is measured, not assumed. Against model2vec-rs reading the f32 original, over 100 133 text segments extracted from real PDF / DOCX / XLSX / EPUB / ODT / ODS / ODP / PPTX files:

• token ids: 0 divergences
• embeddings: worst cosine 0.999994

Retrieval quality on our internal evaluation is unchanged within noise (0.570/0.413 vs 0.571/0.414 for f32).

Caveats

• [UNK] handling follows model2vec-rs exactly, including that it does not drop [UNK] rows for a Unigram tokenizer (it looks for a string unk_token field, which Unigram does not emit). If you consume these artifacts with your own loader and drop [UNK] instead, you will get different vectors on unk-heavy text.
• vocab.fst maps duplicate tokens to their last id, matching how tokenizers builds its token_to_ids map.
• The FST/score/norm layout is ours, not a model2vec standard. Anything reading only model.safetensors + tokenizer.json (including model2vec itself) works with this repo unmodified — the i8 tensor is a standard safetensors dtype.

License

MIT, inherited from the upstream model.