quant-matmul

W4A8 / W8A8 / FP16-storage GEMV-GEMM for quantized LLM inference on CPUs, aarch64-first. Int4 weights in GPTQ/AWQ-compatible group quantization run against per-token INT8 activations. Companions on the Hub: phanerozoic/bitnet-cpu (ternary linears), phanerozoic/cpu-attn (int8 KV attention), phanerozoic/decode-ops (fused glue and sampling).

Per group g of group_size weights along K:

y[m, n] = sa[m] * sum_g s[n, g] * (dot_g(a_m, w_n) - z[n, g] * asum_g[m])

Group dots are exact in int32 (nibble <= 15, |a| <= 127); groups accumulate in fp32 in fixed order, so every dispatch tier produces bit-identical f32 output. Nibbles are shifted and masked in registers against a one-time even/odd split of the activations; no unpack buffer exists.

Chosen once at runtime from CPUID / HWCAP; QM_CPU_ISA (scalar, avx2, avxvnni, neon, neondot) demotes the selection for A/B runs; it never promotes. Every tier produces bit-identical f32 output.

Usage

import torch
from kernels import get_kernel

qm = get_kernel("phanerozoic/quant-matmul", version=1, trust_remote_code=True)

# From a GPTQ checkpoint's tensors:
packed, s, z, gs, perm = qm.from_gptq(qweight, qzeros, scales, g_idx)
y = qm.w4a8_linear(x, packed, s, z, gs, perm)            # [..., N]

# Or as a module:
layer = qm.QuantLinear.from_gptq_tensors(qweight, qzeros, scales, g_idx)
y = layer(x)

version selects the release branch; trust_remote_code is required by kernels for publishers without the trusted-publisher mark.

API

zero_offset=1 follows the AutoGPTQ storage convention (the stored zero plus one is subtracted at dequant); pass 0 for checkpoints without it.

Performance

Raspberry Pi 5 (4x Cortex-A76 2.4 GHz, SDOT path) and Raspberry Pi 4 Model B (4x Cortex-A72 1.8 GHz, NEON path), 64-bit Raspberry Pi OS, torch 2.13 CPU, group_size 128:

Decode (M=1) streams packed weights at 8-12 GB/s on the Pi 5, the board's memory ceiling; M=128 runs 185 GOPS on the SDOT tier.

On x86-64 (12th-gen Intel mobile, AVX-VNNI), W4A8 M=1 4096-square runs 0.16 ms (1.6x its scalar path) and the fp16-storage GEMV 0.25 ms (7x), tracking the host's higher memory bandwidth.

Requirements

• K divisible by group_size; group_size a multiple of 32 (64 and 128 tested).
• bf16 or f32 activations; output follows the input dtype.
• GPTQ act-order (g_idx) is supported through an activation permutation applied per forward.
• Fast paths cover aarch64 (NEON; SDOT with dotprod) and x86-64 (AVX2; AVX-VNNI where present). Anything else uses the correct scalar fallback.

Scope

Hub-loadable quantized linears for transformers-on-CPU, aimed at boards where a dedicated llama.cpp runtime was previously the only usable path. llama.cpp remains faster as a standalone runtime; this pack is for models composed in Python.

License

Apache-2.0.