aquaman164/MiniMax-M3-VQ-2.4bit

MiniMax-M3 — mixed-bit VQ (AQLM) ~2.4 bpw

A 2.4 bits/weight vector-quantized build of MiniMax-M3 (427B-parameter MoE, 57 sparse-MoE layers × 128 experts, top-4 routing), tuned for English / Chinese / Japanese general quality and built to serve on 2× RTX PRO 6000 (sm_120) through official vLLM with long context.

Serving code: github.com/mmzz164/vqmoe · Quantizer: OneCompression

What it is

• Routed experts (413B params, 96.7% of the model): mixed 1 / 2 / 3-bit vector quantization (AQLM-style — per-group codes + shared codebook + GPTQ error compensation). Module bit distribution {1: 6630, 2: 2995, 3: 12263} (3-bit-heavy barbell).
• Non-expert spine (attention / shared experts / dense MLP): symmetric 4-bit RTN. Router, MSA indexer, norms, embeddings, lm_head, and the vision tower stay bf16.
• Size: ~130.2 GiB (46,484 tensors) — 16% of the 796 GiB BF16 original, ~65 GiB/GPU under TP=2.
• Allocation: loss-aware — gxw = output-Fisher × input-energy × quantization error, an Optimal-Brain-Quantization 2nd-order cost — solved to a 2.4 bpw budget.

Quality (KL to BF16, lower = better; fake-quant on held-out corpora)

corpus	KL(BF16 ‖ this build)	PPL BF16 → 2.4-bit
Japanese-reasoning hold-out	0.219	11.51 → 12.14
neutral multilingual (diverse)	0.440	2.90 → 3.84

The BF16 self-consistency check reads KL ≈ 0 on both corpora (harness sanity). The per-sequence KL is tight across all 8 evaluated sequences on each corpus — a distribution-wide result, not a single-domain artifact.

Efficiency vs the prior 3.2-bit AutoRound build of the same model: both incur essentially the same per-token cross-entropy degradation (≈ 0.28 nats/token, i.e. ~+32% PPL), but this build does it at 0.8 fewer bits per weight (2.4 vs 3.2) — same quality loss, ~25% smaller experts. That is the point of VQ + compensation + loss-aware allocation: it holds quality further down the bit curve than scalar quantization, whose error grows super-linearly below ~3 bits.

Behavioural gate (greedy, temp 0): multi-digit arithmetic in JA/EN/ZH answers correctly and terminates (127×8=1016, 347×29=10063, …); JA/EN/ZH coherence stays in-language (no Chinese fallback); needle-in-a-haystack retrieval succeeds at ~7k and ~15k tokens.

Serving (sm_120 / consumer Blackwell)

• Backend: official vLLM 0.23.1 native MiniMax-M3 (MSA lightning indexer, TRITON_ATTN backend) + OneCompression VQ dequant kernels. No sm_120 sparse patch is needed — unlike DSA models, M3's native indexer runs on stock kernels here. Tensor-parallel = 2, expert-parallel on.
• Context: 40,960. KV at gpu_util 0.97 ≈ 372,736 tokens (~27 GiB/GPU free for KV — the small weight footprint leaves generous headroom).
• Throughput: ~7.3 tok/s decode in eager mode. (CUDA-graph capture currently hits an sm_120 race and is disabled; eager is the supported path.)
• Thinking: M3's native <mm:think> blocks. The chat template exposes a thinking toggle — chat_template_kwargs {"thinking_mode": "disabled"} gives direct answers; the default is adaptive. Pair with vLLM's --reasoning-parser minimax_m3 to split reasoning from content.

See vqmoe/models/minimax-m3 for the launcher, the API server, and the checkpoint-key adaptation.

Limitations

• Vision weights are retained (bf16) but the tested/served path is text-only; the image-text path is unverified for this quantized build.
• On casual creative prompts the reasoning trace can fail to self-terminate at this bit-width (an inherent sub-2/3-bit "no-exit" tendency); thinking_mode: "disabled" fully contains it.
• Decode ~7 tok/s (eager): CUDA-graph acceleration is blocked by an sm_120 capture race, not tunable here yet.
• Reproducing the exact serving stack needs official vLLM 0.23.1 (nightly) + the OneCompression VQ kernels + the small adapter in the vqmoe repo — see its "Reproducibility status".

License & provenance

• Base model: MiniMax-M3 under the MiniMax Community License (see LICENSE). This is a quantized derivative and inherits that license. Commercial use requires displaying "Built with MiniMax M3" and, above the revenue threshold in the license, prior authorization from MiniMax — read LICENSE before any commercial deployment.
• Quantizer: OneCompression (MIT, Fujitsu Ltd. + VQ extensions © mmzz164). Serving glue: vqmoe (MIT).