nbeerbower/Inkling-colibri-int4

Inkling — colibri int4

Thinking Machines Inkling (975B total / 41B active MoE) converted to the colibri expert-streaming container, for CPU-first inference on a single machine.

Quantized from the original BF16 checkpoint (not the NVFP4 release), so the routed experts take a single quantization step: bf16 → int4.

Format

• Routed experts (~95% of parameters): int4, symmetric per-row scales. Each ...mlp.experts.gate_up_proj / down_proj tensor is packed U8 (low nibble = even column, offset +8) with a <name>.qs f32 sidecar of row scales — the standard colibri container read by glm.c/inkling.c.
• Attention, dense MLP, shared experts, embeddings, lm_head: bf16 passthrough.
• Norms, short-conv weights, relative-position banks, router (+bias, scales): f32.
• TML fused w13 tensors were de-interleaved (gate/up rows alternate in the source checkpoint) and renamed to the HF layout, matching transformers' conversion mapping.
• Vision/audio encoders and the MTP head are not included (text-only inference).

Total size ≈ 469 GB. Runtime RSS ≈ 90 GB (bf16 residents) + expert cache.

Usage

Requires the inkling branch of nbeerbower/colibri (Stage-A engine; upstreaming planned):

git clone -b inkling https://github.com/nbeerbower/colibri
cd colibri/c && make inkling
SNAP=/path/to/this/repo ./inkling -p "The capital of France is" -n 64

The engine validates bit-exact against HF transformers on a tiny oracle (tools/make_tiny_inkling.py), and this snapshot was produced by tools/convert_inkling_int4.py (threaded, resumable, shard-at-a-time).

Caveats

• Stage-A performance: no expert pinning/prefetch yet — cold decode is slow (expert streaming bound). Warm-cache and GPU-tier work is in progress.
• No quality evals yet beyond greedy smoke tests; int4 per-row on the experts is the same recipe colibri uses for GLM-5.2 744B.
• Tokenizer is o200k-family; the branch's tok.h handles it natively (357/357 encode-exact vs HF tokenizers in testing).