code-daemon-embed-v1

A 46.8M-parameter, 4-layer code-embedding model that maps short code units — function and method bodies, signatures, docstrings, symbol names — and short natural-language queries into a shared 768-dim space. INT8, 128-token window, 1.9k-5.5k texts/s in ordinary use on a laptop RTX 5060, 10.7k sustained on a large corpus.

It is built for one job: embedding a whole repository fast enough to re-index it on every commit, so that a coding agent can run semantic search on every question it is asked. Every trade-off below follows from that — the depth, the 22.7k vocabulary, the 128-token cap, the INT8 weights, the four length-bucketed engines.

# the whole API surface — pooled AND L2-normalized inside the graph
vec = session.run(None, {"input_ids": ids, "attention_mask": mask})[0]  # [B, 768], ready

Queries and documents are embedded the same way — no query: / passage: prefix.

Pick this model for throughput, not for accuracy — and for English. Against every public 768-dimension encoder tested it is 4× to 6× faster and 18 to 28 nDCG behind on our own held-out queries. Its vocabulary was pruned to code and English, so Russian queries score 0 of 34 where English ones score 9. What it gives is INT8 compiled engines, a pre-pooled graph, a hard 128-token cost ceiling and 8k-14k texts/s in a full index, as the dense half of a hybrid retriever. Numbers and method: Against 768-dimension encoders.

---

1. What it is for, and why not a standard embedder

The target workload

1. A repository is indexed: every function, method, type and doc chunk becomes one short text and one vector. A 700k-entity C++ codebase embeds in ~81 seconds on one consumer GPU.
2. Someone asks something short and keyword-shaped — "git watcher head change reindex", "acquire database lock for project hash", "where does the daemon start".
3. The vector channel runs next to a lexical (BM25-style) channel and the two are fused.

Point 3 matters: this model was trained to be the dense half of a hybrid retriever, not to win alone. Its training queries follow the shape an agent actually issues — short keyword bags, behaviour descriptions, identifier fragments — rather than docstring paraphrases.

Design choices vs a typical general-purpose text embedder

Against the e5 / bge / gte class:

aspect	typical	this model
Parameters	110M – 7B	46.8M
Vocabulary	30k – 250k	22.7k, code-pruned
Max sequence	512 – 8192	128, hard cap
Query format	instruction prefix	none, symmetric
Pooling	you implement it	in the graph
Weights	FP32 / FP16	INT8, QAT
Top-rank precision	separate reranker pass	distilled into the vectors

Ranking distilled from a cross-encoder

A bi-encoder compresses each document into a vector before it sees the query, so it cannot do the pairwise comparison a cross-encoder reranker does. This model closes part of that gap at training time instead of inference time: Qwen/Qwen3-Reranker-4B scored every (query, candidate) pair over mined hard negatives, and the student was trained with a listwise-KL objective to reproduce the teacher's ranking distribution — not merely "positive above negative", but how much each near-miss should trail.

Practical consequence: you probably do not need a runtime reranker on top of it. In the production system it was built for, switching one on measured net-negative — the distillation had already captured the useful part, at zero inference cost.

Strong / weak

Strong

• Repository search with short NL, keyword and identifier queries.
• Natural language → code at short lengths.
• The dense channel of a hybrid retriever.
• Throughput-bound work: bulk re-index, index-on-save, re-embed per commit.

Weak / out of scope

• Long documents. Hard 128-token cap. Not a long-context retriever (see the window recipe in §4.6 if you must handle longer text).
• Benchmark-style long problem statements, multi-turn dialogue, code↔code translation.
• General English prose (medical / financial / news) — the pruned vocabulary trades that away deliberately.
• Non-English queries. The prune kept 942 Cyrillic pieces of XLM-R's 31 671; Russian queries about code score 0 of 34 on a set where the same queries in English score 9. The multilingual-e5-base backbone this started from handles them; this model does not.

---

2. Architecture — what was kept, cut and added

Specification

Property	Value
Encoder layers	4
Hidden size	768
Attention heads	12
FFN	3072, GELU
LayerNorm eps	1e-5
Vocabulary	22,739
Position embeddings	514
Max sequence	128
Parameters	46.80M

• Base — intfloat/multilingual-e5-base (XLM-RoBERTa encoder, 12L, 278M).
• Vocabulary — SentencePiece unigram pieces with byte fallback; positions are learned and absolute. Special ids pad=0, unk=1, bos=2, eos=3 (raw-SentencePiece indexing).
• Parameter split — 17.46M embedding table + 4 × ~7.3M encoder + positions.
• Output — 768-dim, mask-mean-pooled AND L2-normalized inside the graph. Use it as it comes.
• Weights — INT8 from quantization-aware training; the Q/DQ nodes carry the trained scales.
• ONNX — opset 19; inputs input_ids, attention_mask, both int64 [batch, seq].

Removed from the base model

• 8 of 12 encoder layers — strided-truncated 12 → 8 → 6 → 4, healed after each cut.
• 227k of 250k vocabulary pieces: 250,002 → 22,739, and the embedding table with them (192M → 17.5M parameters — the single largest saving in the model).
• The E5 instruction prefixes. No query: / passage: asymmetry; both sides are encoded identically.
• The pooler head, the MLM head, token-type inputs. The graph has exactly two inputs.
• FP32 weights — replaced by INT8 with trained scales.

Added

• Mask-aware mean pooling and L2 normalization fused into the ONNX graph. The model returns unit-norm [B, 768], not [B, seq, 768] — there is no pooling code to get wrong, no last_hidden_state copy, and no way to accidentally compare unnormalized vectors.
• Q/DQ nodes with QAT-trained scales, so a TensorRT/OpenVINO build is INT8 end-to-end without a calibration pass.
• Ranking knowledge from a 4B cross-encoder (the rank-distillation above).
• Four length-bucketed engine builds with a dynamic sequence dimension (§3, §4.4).

How it was made, in brief

The e5 backbone was truncated in depth, its vocabulary pruned and re-indexed to the raw-SentencePiece id convention, then trained by distillation — dense representation distillation against the embedding teacher, plus listwise-KL ranking distillation against the cross-encoder teacher, which is where the rank knowledge in section 1 comes from. The INT8 weights come from quantization-aware training, not from a calibration pass, and the export carries the scales learned during it.

⚠ If you build your own engines from this ONNX

Do not run PTQ/calibration over the shipped INT8 graph. It overwrites the trained scales with fitted ones and measurably degrades the model. model_int8qdt.onnx is the artifact — feed it to your builder as-is (sections 4.4 and 4.5 show the TensorRT and OpenVINO invocations). This is the failure mode that produces an engine which builds cleanly, loads cleanly and quietly retrieves worse.

---

3. Performance

Measured on one RTX 5060 Laptop GPU (sm_120, 8 GB), TensorRT INT8, pinned TDP, clocks.sm 2,647 MHz median at 114 W.

End-to-end, inside the production indexer

The whole path: build the serve text → tokenize on host threads → IPC to the worker → GPU forward → pool → collect.

corpus	vectors	wall	pipeline texts/s	GPU-only texts/s
mysql-server (sustained, 54 batches)	868,795	80.96 s	10,731	11,712

⚠ That is a best case, and a two-device one. Re-measured 2026-09-11 on the same corpus: 10 686 texts/s pipeline, but only 5 680 texts/s of model forward — the daemon adds the Intel iGPU (OpenVINO) alongside the RTX 5060 once a batch exceeds ~820 000 texts, and mysql-server does. A full index of a smaller repository runs on one device, and the rate then depends on the language: 13.8k texts/s for C/C++, 10.6k for TypeScript, 8.0k for Java, 7.9k for C# (see In a real index, by programming language).

Small incremental runs are slower again — 1 900 - 5 500 texts/s for deltas of a few hundred to fifteen thousand vectors — because tokenisation and IPC are paid per batch and there are few batches to amortise them over. Size a deployment for your workload, not from the headline.

pipeline texts/s = vectors ÷ wall clock, everything included. GPU-only = vectors ÷ summed inference time; it is higher because host work overlaps with the GPU.

Solo engine profiles — cost per sequence length

Four engines, one per length bucket, each with a dynamic sequence dimension so a batch can be dispatched at its own longest text instead of the bucket ceiling. One engine alone, dispatch only, 200 iterations:

s — 96 × 8…48, opt 48

seq	texts/s
8	72,587
16	50,681
24	37,228
32	29,651
40	23,495
48	19,785

m — 128 × 56…64, opt 56 · l — 128 × 72…80, opt 72

seq	texts/s
56	15,853
64	13,068
72	11,302
80	9,913

xl — 256 × 88…128, opt 96

seq	texts/s
88	8,537
96	7,816
104	7,027
112	6,421
120	5,907
128	5,464

Three things to take from that table when you build your own serving path:

• Cost is per token, not per text, and it is superlinear: solo token throughput falls from 1.01M tok/s at seq 48 to 0.71M at seq 128. Batch size is not the lever — a 256×64 engine measured 5.8% slower per text than 128×64, because 96×48 already saturates the SMs.
• Padding is the lever. A batch pays for its longest member, so sorting texts by length before batching — and dispatching each batch at its own length — was worth +10.2% end-to-end. Round lengths up to a multiple of 8: a non-multiple is slower than a longer multiple (seq 70 costs more than seq 72).
• Running the four engines concurrently does not add throughput on one GPU. In parallel they measured slower than serially (1,487 ms vs 1,423 ms for the same work). The lanes exist to keep the GPU fed while the host works, not to multiply throughput.

Without a discrete GPU — OpenVINO on CPU, integrated GPU and NPU

Same model, no NVIDIA card involved. Measured on an Intel Core Ultra 9 275HX — its CPU cores, its integrated GPU, and its AI Boost NPU (13 TOPS) — with OpenVINO 2026.3 and Intel NPU driver 32.0.100.4841. Both versions matter for the NPU rows: the numbers moved by an order of magnitude across an earlier OpenVINO release, so treat them as version-stamped. The IRs in this repository are built for OpenVINO 2026.4; the same IRs timed under both runtimes on the same machine (default hints, one synchronous request, interleaved rounds) put 2026.4 within 2–5 % slower on the CPU — inside this machine's round-to-round spread — and 0–4 % faster on the iGPU and the NPU. An OpenVINO IR is reshaped to one static shape at build time (unlike the TensorRT engines, whose sequence dimension is a range), so each bucket is a single number per device:

Texts embedded per second, per bucket:

bucket	CPU INT8	iGPU INT8	NPU INT4
s	702	1,092	565
m	509	847	430
l	402	678	298
xl	237	420	169

Shapes: CPU and iGPU run batch 64, the NPU batch 16; seq 48 / 64 / 80 / 128 for s / m / l / xl.

All three at once. These are solo numbers — one device, nothing else running. The three share one LPDDR5 controller, so they do not simply add up, but they come close: measured in a single shared window at 64 × 64, CPU 416 + iGPU 750 + NPU 377 = 1,543 against 527 + 822 + 423 = 1,772 solo, i.e. 87 % of the sum. The noisiest neighbour is also the fastest device (the iGPU costs the others ~13 %, the CPU ~7 %), so there is nobody worth switching off.

End to end that lands at ~1,050 texts/s on a real 20k-entity repository — the daemon runs all three workers concurrently and that figure includes tokenization, IPC and vector writes, not just inference.

Reference point: the same buckets on the RTX 5060 run at 19,785 / 13,068 / 9,913 / 5,464 texts/s, so the integrated GPU is 15–18× slower than the discrete one. That still leaves a 100k-entity repository indexed in a few minutes on a machine with no dedicated GPU at all.

Three notes if you deploy this path:

• INT8 on CPU and iGPU, INT4 only on the NPU. At 128 × 64 the iGPU does 859 texts/s in INT8, 584 in FP16 — and 584 in INT4, exactly the FP16 number. INT4 here is weight-only compression: the weights decompress to fp16 before the GEMM, so it is the same GEMM and the 4 bits buy file size, not speed. Only INT8, which the kernels execute natively, moves the number. The NPU is the exception because it requires INT4 with static shapes.
• Feed it the Q/DQ graph, not a calibrated one. OpenVINO keeps the QAT scales as FakeQuantize; re-fitting them with a post-training pass measured hit@1 .200 → .133 on this model. See §4.5 for the two-line recipe (and the transformation that keeps the IR at 96 MB instead of 179 MB).
• The NPU pays for batch, not for tokens. Its IRs are baked at batch 16 against 96–256 elsewhere, which is why its per-text cost is the flattest across buckets and its per-batch latency the lowest (28–95 ms). It suits interactive single-query embedding better than bulk indexing.

Retrieval quality

80 captured real agent queries over a live indexed codebase, multi-positive, file-level, through a production hybrid retriever (dense + lexical, runtime reranker off):

metric	value
hit@1	0.46
hit@3	0.74
hit@5	0.78
hit@10	0.80
mrr@10	0.59
ndcg@10	0.61

This is the metric the model is optimized for. On a broader 2,441-query gate across four unrelated repositories, the 4-layer model trades about 4% relative hit@1 against its 6-layer parent for roughly 1.5× the engine speed.

CoIR — the out-of-domain reference

Run on this exact artifact — model_int8qdt.onnx, full corpora (2.3M documents across the six tasks), NDCG@10, the same raw-SentencePiece tokenization the daemon uses:

CoIR task	NDCG@10	Pattern
synthetic-text2sql	51.26	NL → SQL
stackoverflow-qa	41.63	short question → code
codesearchnet (6-lang avg)	39.02	docstring / NL → code
codefeedback-st	38.55	NL instruction → code
codesearchnet-ccr (6-lang avg)	36.07	code → related code
cosqa	17.50	NL question → code (noisy / hard)
Average	37.34

Per language — codesearchnet (NL→code): python 68.30, go 46.15, java 34.64, ruby 28.76, js 28.33, php 27.95. codesearchnet-ccr (code→code): js 43.36, ruby 42.88, java 36.82, php 31.59, go 31.06, python 30.68.

Read this as a lower bound, not as the headline. CoIR queries are mostly docstrings and long problem statements — the opposite of what this model was tuned for, and the spread proves it: 68.30 on Python docstring→code against 17.50 on cosqa's noisy question→code. Four of CoIR's ten tasks (code↔code translation, multi-turn dialogue, long problem statements) exceed the 128-token scope and are not shown. A small share of out-of-domain retrieval pairs was present in training as diversity regularization; overlap with the test splits was not audited.

Against 768-dimension encoders — read this before choosing the model

Every number above compares this model to its own earlier versions. That answers did it improve, not is it the right pick. Measured 2026-09-11 on one laptop, vector-only cosine (no BM25, no fusion, no reranker), every model at the same 128-token budget.

Only 768-dimension models are compared. A 384-dimension encoder is cheaper in the forward pass, in the index and at query time — a different product, not a faster one — so putting it in the same column would be a category error.

Accuracy, nDCG@10 × 100 on one scale. Three CoIR tasks chosen for the queries this model serves (short question → code, code → code), and our own held-out golden set:

model	cosqa	stack-qa	code→code	our golden
code-daemon-embed-v1 (INT8)	18.5	41.7	30.7	42.7
CodeRankEmbed	35.5	73.3	—	64.6
multilingual-e5-base	29.8	79.4	57.7	60.7
gte-modernbert-base (teacher)	36.6	81.4	82.5	70.5

codesearchnet (docstring → code) is left out because it matches neither query the daemon issues — and it is the task this model scores highest on (68.3), so dropping it makes the table less flattering, not more.

Speed, texts/s through one pipe — every model exported to ONNX with pooling baked in, run under onnxruntime CUDA at FP32, batch 64, minimum of five rounds, across the shipped bucket lengths. The mean serving text is 69.6 tokens, so the middle columns are the working point:

model	layers	40	64	80	128
code-daemon-embed-v1	4	5 392	3 457	2 644	1 457
multilingual-e5-base	12	1 355	854	666	385
CodeRankEmbed	12	1 249	723	572	338
gte-modernbert-base	22	1 093	606	446	243

4.0× multilingual-e5-base, 4.3-4.8× CodeRankEmbed, 4.9-6.0× gte-modernbert-base. At equal width the layer count predicts it — 4 against 12 is 3×, 4 against 22 is 5.5× — and the residual above that is the embedding table: 17.5 M rows against 192 M is a gather with far better locality. The shipped INT8 TensorRT engine is faster again (5 464 texts/s at seq 128).

So the trade, stated once: four to six times the throughput of any public 768-d encoder, for 18 to 28 points of nDCG on our own queries. The teacher is 28 ahead of its student — that is the distillation gap, not a benchmark artefact.

Language

The backbone is multilingual; this model is not. The 34 held-out queries translated to Russian, same corpus, same gold, identical text for every model:

model	EN hit@1	RU hit@1	RU hit@5
multilingual-e5-base	0.382	0.235	0.529
gte-modernbert-base	0.559	0.147	0.294
CodeRankEmbed	0.412	0.000	0.235
code-daemon-embed-v1	0.265	0.000	0.029

The cause is countable: the 250k → 22.7k vocabulary prune kept 942 Cyrillic pieces against XLM-R's 31 671, so Russian fragments into byte-level pieces. n = 34 and the translations are ours (the same text goes to every model), and this is the dense channel alone — in a hybrid retriever a Russian query carrying English technical terms will still partly hit lexically. If your queries are not in English, use multilingual-e5-base.

In a real index, by programming language

A full --reset index of one large repository per language through the UltraCode daemon, read from its own log. Same engine, same laptop:

language	repository	entities	xl bucket	texts/s
C/C++	mysql-server	1 144 040	24 %	13 821
TypeScript	vscode	478 925	44 %	10 644
Java	netty	96 547	69 %	8 006
C#	roslyn	403 494	71 %	7 930

Throughput depends on the language, not just the model. C# entity texts are long — 71 % land in the most expensive (seq 88-128) engine — while C/C++ ones are short and repeat, 23 % of them served free from in-batch deduplication. The 1.7× spread between the top and bottom rows is text length, not hardware.

The 10.7k in §3 is the mysql-server row, and it is a two-device number: above ~820 000 texts the daemon adds the Intel iGPU (OpenVINO) alongside the RTX 5060 (TensorRT). The model's own forward rate on that run was 5 680 texts/s; the other three repositories stayed below the threshold and ran on one device.

The same comparison inside the product. multilingual-e5-base — this model's own ancestor — compiled to the same TensorRT INT8 and OpenVINO engines, the same bucket grid, selected by config and run through the same full index:

language	this model	e5-base	ratio
C/C++	13 821	5 758	2.40×
TypeScript	10 644	4 620	2.30×
Java	8 006	3 562	2.25×
C#	7 930	3 435	2.31×

texts/s. The gap is 2.3× here against 4× through onnxruntime FP32, and the engines say why: e5's quantized graph keeps attention in BF16, where TensorRT fuses it into one kernel, while this model's QAT graph keeps attention in FP32, where it cannot. If you compile e5 yourself, check the engine against its FP32 graph — a default INT8 build came out at cosine 0.69 and reported success.

Harness, raw JSON and the full write-up ship in the UltraCode repo (models/_distill_shared/bench_vs_generic.py, .autodoc/benchmarks.md). The harness was validated first by reproducing two numbers from this card — stackoverflow-qa 41.63 → 41.65 and codesearchnet python 68.30 → 68.32.

4. Using the model

4.1 What is in this repository

Source of truth — always present:

• model_int8qdt.onnx — INT8 Q/DQ graph with QAT-trained scales. Every TensorRT and OpenVINO INT8 engine is built from this file.
• model.onnx — the FP32 twin of the same weights, for lanes that cannot read Q/DQ and for fine-tuning. Derived from the file above by dropping its Q/DQ pairs, so the weights are identical — the 0.98 cosine between them is the quantization, not a different model.
• model.safetensors + config.json — the same float weights under transformers names, exported from model.onnx and checked against it (max |Δ| 3e-7 on the pooled vector). AutoModel loads the encoder (XLMRobertaModel); the trained projection proj.weight is not part of that class, so apply it yourself: mean-pool over the attention mask → proj → L2. config.json → ultracode spells the head out. This pair is also what the Apple (MLX) build is prepared from.
• sentencepiece.bpe.model — the tokenizer. Raw-SP ids: pad=0, unk=1, bos=2, eos=3. Its 22,739 pieces match the embedding table row for row.
• tokenizer_config.json — HF-side metadata.

Prebuilt engines, uploaded progressively per architecture:

TensorRT    …-{s,m,l,xl}_{win_x64,linux_x64}_trt11.0_sm_{75,80,86,89,120}.engine
            …-{s,m,l,xl}_linux_x64_trt11.0_sm_90.engine
OpenVINO    …-{s,m,l,xl}_ov2026.4_{cpu_int8,igpu_lnl_int8,npu_int4}_b*_s*.{xml,bin}
TVM Vulkan  …_{win_x64,linux_x64}_tvm0.25_vulkan.{dll,so}
MLX (Apple) model_gpu_mlx0.32/

sm_90 (H100 / H200) is Linux-only. An INT4 IR for discrete Intel Arc (igpu_arc_int4) is declared by the daemon but not built yet.

If the engine you want is missing, build it from the ONNX — §4.4.

4.2 Taking a prebuilt engine

The filenames are a loading contract, not decoration. A serialized TensorRT plan is keyed on {GPU architecture × OS × TensorRT version} and deserializeEngine has no compatibility fallback, so pick all four coordinates exactly:

code-daemon-embed-v1-m_win_x64_trt11.0_sm_120.engine
                     │  │       │       └── GPU arch: sm_75 Turing · sm_80 Ampere (A100, A30)
                     │  │       │           sm_86 Ampere (RTX 30xx, A-series) · sm_89 Ada (RTX 40xx, L4)
                     │  │       │           sm_90 Hopper (H100, H200) · sm_120 Blackwell (RTX 50xx)
                     │  │       └── TensorRT 11.0 — not interchangeable with 10.x
                     │  └── OS/ABI
                     └── length bucket: s | m | l | xl

Bucket shapes (batch × seq): s 96×48, m 128×64, l 128×80, xl 256×128. Loading all four costs ~400 MB of VRAM; if you only want one, take m — it covers the fattest part of a typical corpus (25–40% of texts land in 49…64 tokens).

Route each text to the first bucket whose sequence ceiling fits its token count, and pad the batch to that bucket's shape. Padding is masked in attention, so a padded batch and an unpadded one give the same vector (verified at cosine 1.000000) — provided the mask is right.

4.3 Running the ONNX directly

Works anywhere ONNX Runtime does (CPU, CUDA, DirectML) with no build step:

import onnxruntime as ort, sentencepiece as spm, numpy as np

sp   = spm.SentencePieceProcessor(model_file="sentencepiece.bpe.model")  # pad=0 unk=1 bos=2 eos=3
sess = ort.InferenceSession("model_int8qdt.onnx", providers=["CPUExecutionProvider"])

def embed(texts, max_len=128):
    ids  = [[2, *sp.encode(t)[: max_len - 2], 3] for t in texts]            # bos … eos
    L    = max(len(x) for x in ids)
    inp  = np.array([x + [0] * (L - len(x)) for x in ids], dtype=np.int64)  # pad=0
    mask = (inp != 0).astype(np.int64)
    # pooled and unit-norm already — no post-processing
    return sess.run(None, {"input_ids": inp, "attention_mask": mask})[0]    # [B, 768]

D = embed(["function acquireLock in src/db.zig: zig\npath: src db"])
Q = embed(["acquire database lock"])
print(Q @ D.T)          # inner product IS cosine here

Two notes that cost real debugging time:

• Tokenize with SentencePiece, not a greedy-BPE merge loop. The vocabulary is a unigram model; scoring it with pair-merge BPE produces a different segmentation than training used, and that train/serve skew silently costs retrieval quality.
• Pad with id 0 and mask 0. Both inputs are int64; feeding int32 buffers to a hand-written runtime is the classic "second half of every batch is garbage" bug.

4.4 Building your own TensorRT engines

TensorRT 11 reads precision entirely from the ONNX Q/DQ nodes — the per-precision flags (--int8, --fp16, --calib) were removed. Feed it the Q/DQ graph and pass --stronglyTyped. Feed it an unquantized graph instead and you get an FP16 engine that builds cleanly, loads cleanly, is twice the size and a fraction of the speed — the size is the only visible symptom, so check it.

Rectangular engine for one bucket (here m, 128×64):

trtexec --onnx=model_int8qdt.onnx \
        --saveEngine=code-daemon-embed-v1-m.engine \
        --stronglyTyped \
        --builderOptimizationLevel=5 \
        --timingCacheFile=timing.cache \
        --minShapes=input_ids:1x1,attention_mask:1x1 \
        --optShapes=input_ids:128x64,attention_mask:128x64 \
        --maxShapes=input_ids:128x64,attention_mask:128x64

Dynamic-sequence engines (the shipped configuration, +10.2% end-to-end) keep the batch dimension rectangular and let seq range. Use these profiles:

bucket	batch	seq min	seq opt	seq max
s	96	8	48	48
m	128	56	56	64
l	128	72	72	80
xl	256	88	96	128

Three rules behind those numbers:

• One dynamic engine per bucket, opt at that bucket's mean length — never one global dynamic engine. The tax is paid for being far from the opt point, not for being dynamic: a single wide (16→64→128) engine costs +0.8% at seq 64 but +15.4% at seq 80.
• Sequence length must be a multiple of 8 at dispatch (seq 70 costs more than seq 72).
• Sort texts by length within a bucket and cut at batch boundaries before dispatch. Without the sort the dynamic engines buy ~0%: any 128-text batch drawn from the 49–64 range almost surely contains a 64 and dispatches at 64 anyway.

A dynamic build reports ~69 layers against a rectangular build's ~54 — a wider shape range costs fusions, and that is exactly where the small tax at the ceiling lives.

4.5 OpenVINO, TVM, MLX

OpenVINO reads the Q/DQ graph directly — the trained scales survive as FakeQuantize, so there is no calibration pass to run:

import openvino as ov
from openvino._offline_transformations import compress_quantize_weights_transformation

m = ov.Core().read_model("model_int8qdt.onnx")
compress_quantize_weights_transformation(m)                       # folds weights to i8: 179 MB -> 96 MB
m.reshape({"input_ids": [128, 64], "attention_mask": [128, 64]})  # one bucket, static
ov.save_model(m, "code-daemon-embed-v1-m_cpu.xml")

That transformation is not optional bookkeeping: read_model leaves every weight as f32 behind a FakeQuantize, and the IR comes out nearly twice the size for the same arithmetic. Do not reach for NNCF post-training quantization here — it would replace the trained scales with fitted ones, which is the failure the warning in §2 describes.

The NPU artifacts are the exception: they start from the FP32 twin and apply INT4 weight compression at batch 16, trading accuracy for size on purpose. TVM Vulkan modules and the MLX weights are also built from the FP32 twin, per bucket.

4.6 Feeding it well

Documents. The model was trained on a compact, front-loaded "serve text": semantics first, identifiers after, everything inside the 128-token budget. Reproducing that shape on your own corpus is worth more than any inference tuning:

{type} {name} in {file}: {lang}
path: {directory tokens, space-separated}
[async] [exported] [test]
sig: ({params}) -> {return type}
{doc comment, first ~200 chars}
{one-to-two-sentence description}

for example:

function acquireProjectLock in src/storage/multi_db.zig: zig
path: src storage multi db
[exported]
sig: (allocator: Allocator, project_hash: []const u8) -> !Lock
Acquires the exclusive SQLite lock for one project.

Cap the whole text around 768 characters; the raw function body is deliberately not part of it (it belongs in the lexical channel, where it measurably helps, not in the vector).

Queries. Feed them raw, no prefix, no template. The model is tuned for short keyword bags and behaviour descriptions.

Text longer than 128 tokens. Split into overlapping windows — window 128, stride 96 — embed each, then mean-pool the window vectors and L2-renormalize. That is what the production indexer does for long doc chunks.

Retrieval. The vectors come out unit-norm, so inner product is cosine — no normalization step of your own. They are dense and 768-dimensional; an IVF/HNSW index over them behaves normally.

---

License

Released under the MIT license.

The backbone (intfloat/multilingual-e5-base) is MIT; the teachers (gte-modernbert-base, Qwen3-Reranker-4B) are Apache-2.0. As is standard practice for distilled embedding models, the weights are released under MIT. Not legal advice.

Attribution

Backbone: intfloat/multilingual-e5-base (MIT). Dense teacher: Alibaba-NLP/gte-modernbert-base (Apache-2.0). Ranking teacher: Qwen/Qwen3-Reranker-4B (Apache-2.0).