gguf-gemv

GEMV directly on packed GGUF blocks, aarch64-first. Weight rows stream as raw block bytes; each block decodes to f32 in a stack buffer and accumulates against the activations, so no dequantized weight tensor is ever materialized. This runs GGUF-quantized models inside transformers on CPU without the loader expanding a 4-bit checkpoint to fp first. Companions on the Hub: phanerozoic/bitnet-cpu, phanerozoic/quant-matmul (linears), phanerozoic/cpu-attn (int8 KV attention), phanerozoic/decode-ops (fused glue).

Supported ggml block types: Q4_0, Q8_0, Q4_K, Q5_K, Q6_K (ids 2, 8, 12, 13, 14), the mainline ternary TQ2_0 (35), and the BitNet.cpp ternary I2_S (36). The first six cover the common K-quant files (Q4_K_M, Q5_K_M) and reproduce the ggml / gguf-py reference dequantization bit for bit (max abs difference 0 on aarch64 and x86-64). I2_S follows the BitNet.cpp quantize_i2_s / vec_dot_i2_i8_s layout (128-element blocks, one f32 tensor-wide scale, codes {0,1,2} -> {-1,0,+1}), verified bit-exact against that spec and against the packed tensors of microsoft/bitnet-embedding-0.6b. This makes BitNet ternary GGUFs, the embedding and generative models distributed for BitNet.cpp, loadable through the Python stack.

Usage

import torch
from kernels import get_kernel

gg = get_kernel("phanerozoic/gguf-gemv", version=1, trust_remote_code=True)

tensors = gg.load_gguf("model.gguf")       # quantized 2D -> GGUFLinear
layer = tensors["blk.0.ffn_up.weight"]     # GGUFLinear
y = layer(x)                               # [..., N]

load_gguf reads tensor data through the gguf package's copy-on-write mmap, so weight bytes stay file-backed and the kernel reads them out of the page cache. version selects the release branch; trust_remote_code is required by kernels for publishers without the trusted-publisher mark.

API

Performance

Raspberry Pi 5 (4x Cortex-A76 2.4 GHz) and Raspberry Pi 4 Model B (4x Cortex-A72 1.8 GHz), 64-bit Raspberry Pi OS, torch 2.13 CPU:

Decode (M=1) is memory-bound at the block-byte stream; Q6_K is slower for its heavier per-block bit assembly. Correctness is decode exactness against gguf-py; speed is a decode-on-the-fly GEMV, not a repacked-format kernel.

Requirements

• K a multiple of the block size (32 for Q4_0/Q8_0, 256 for the K-quants and TQ2_0, 128 for I2_S).
• f32 or bf16 activations; output follows the input dtype.
• load_gguf reads every type from the raw file, including I2_S, which gguf-py's own reader rejects; it needs no external package.
• The block decode is scalar on both aarch64 and x86-64 (the f32 accumulate vectorizes); correct and bit-exact, not micro-tuned.

Scope

The zoo path: the GGUF catalogue is the largest quantized-model library in existence, and this kernel makes those files loadable for transformers-on-CPU without conversion, now including the BitNet ternary formats (TQ2_0 and the BitNet.cpp I2_S) that carry the BitNet embedding and generative models. A dedicated llama.cpp or BitNet.cpp runtime remains faster; this is for models composed in Python. A future version may add integer-domain block GEMV for the ternary types.

License

Apache-2.0.