visible-cx/gemma4-26b-a4b-CoreAI

Gemma 4 26B-A4B (MoE) — Core AI (.aimodel)

google/gemma-4-26B-A4B-it-qat-q4_0-unquantized converted to a Core AI .aimodel bundle for Apple silicon by visible-cx. This is a derivative artifact: Google's QAT-trained weights rounded onto int4 and re-expressed as a Core AI graph, with the 128-expert sparse branch of every layer lowered onto a Metal gather kernel.

It is a sparse mixture-of-experts model — 26.5B total parameters, roughly 4B active per token.

⚠️ UNQUALIFIED. This bundle has never produced a token, on any machine. The graph exports and the producer fingerprint passes; the block wiring is verified against the reference implementation (below). Nothing else is established.

⚠️ The 16 GB tier is excluded. Weights are ~17.6 GB resident against a 16 GB Mac's ~10.7 GB Metal working set. Minimum practical machine memory: 24–32 GB.

⚠️ Expected known issue — long unguided generations (unverified on this model). On the sibling E2B/E4B exports, the Core AI runtime retains one compiled specialization per sequence-length signature, costing roughly 80 MB of GPU allocations per generated token until process exit; the decode graphs in this repository share the length-dependent structure that causes it. Until a shape-stable re-export exists for this model, treat these bundles as suitable for guided/constrained and short generations; budget memory for long free-form runs accordingly. Measured details: the gemma-4-E2B-CoreAI card's known-issue section.

Provenance

Base checkpoint	google/gemma-4-26B-A4B-it-qat-q4_0-unquantized @ f1e06dc520982d9b9edd76859fdb7ab209449949
Checkpoint size	51,644,341,801 B, bf16, 1,013 tensors across 2 shards; ungated
Recipe	export_gemma4_moe_decode_pipelined.py int4lin --lin-sym --metal-sdpa --max-ctx 16384
Toolchain base	apple/coreai-models @ b1cb71b8522d99408059fa0b98b8742171bcb0b8 + the coreai-model-zoo python overlay, plus three new modules (below)
Toolchain	coreai-torch 0.4.1, coreai-core 1.0.0b2, coreai-opt 0.2.1, torch 2.9.0
Producer fingerprint	coreai-core 1.0.0b2 — verified on the inner asset metadata.json
Asset creation date	20260817T215629Z
Vocab	262,144
Export functions	main (S=1 decode) only

"QAT-unquantized" means QAT-trained, stored bf16; the int4 rounding happens at export onto the ggml q4_0 grid the training already targeted. There is no separate pre-quantised int4 artifact to prefer.

To reproduce this conversion you need a ≥128 GB host: the export peaks at 102.25 GB RSS (VmHWM) on Linux x86_64.

Architecture

	26B-A4B	31B	E4B
hidden_size	2816	5376	2560
layers	30 (25 sliding / 5 full)	60	42
dense MLP intermediate_size	2112	21504	10240
num_experts / top_k	128 / 8	—	—
moe_intermediate_size	704	—	—
attention heads	16	32	8
KV heads (sliding / full)	8 / 2	16 / 4	2 / —
head_dim / global_head_dim	256 / 512	256 / 512	256 / 512
sliding_window	1024	1024	512
Per-Layer Embeddings	none	none	256
KV-shared layers	0	0	18

Same attention family as the dense 12B/31B — same dual head_dim, same attention_k_eq_v (full layers carry no v_proj), same dual RoPE, same softcap. There are no Per-Layer Embeddings and therefore no gather-table sidecar in this repo.

Every layer carries a dense MLP and a sparse branch, in parallel:

residual = x                                              # post-attention hidden
h  = mlp(pre_feedforward_layernorm(x))                    # dense branch
h1 = post_feedforward_layernorm_1(h)
w, idx = router(residual)                                 # routes on the RAW residual
h2 = experts(pre_feedforward_layernorm_2(residual), idx)  # sparse branch
h2 = post_feedforward_layernorm_2(sum_k w_k * h2_k)
x  = residual + post_feedforward_layernorm(h1 + h2)
x  = x * layer_scalar

Two details are load-bearing: the router reads the un-normalised residual, and the experts are GELU-gated (gelu_pytorch_tanh), where the SwitchGLU primitive's default is SiLU. The router itself: scale-free RMSNorm → × scale × hidden**-0.5 → linear to 128 → fp32 softmax over all experts → top-8 → renormalise to sum 1 → multiply by a learned per_expert_scale gathered at the selected ids.

Compression recipe

Weights	Scheme	Rationale
Routed experts (128/layer, 22.84 G params)	affine int4 (aff4), per-(output row, K-block-32) scale + bias, MSE-optimal clip over 6 candidates	see below
Attention, dense MLP, untied LM head	linear int4 per-block-32, plain absmax (--lin-sym)	the ggml q4_0 grid the QAT checkpoint was trained on — the 12B/31B recipe verbatim
Router (proj, scale, per_expert_scale)	fp16, excluded by name	routers are the quantization-sensitive part of an MoE and the whole set is 0.02 GB
embed_tokens (in-graph)	fp16	the gather stays exact; the head is untied and quantized separately

sym8 on the experts would put them at 22.8 GB before anything else, so int4 is not optional. aff4 was chosen over a k-means palette (km4) because this checkpoint was QAT-trained on per-block-32 q4_0 (w = (q-8)·d, 16 uniform levels), and an affine int4 block-32 grid represents that exactly (scale = d, bias = -8d). A palette fits one 16-entry codebook per 32 output rows across the whole K axis and structurally cannot follow per-block QAT scales. The price is aux bytes: aff4 stores an fp16 scale and bias per 32-element K block, i.e. 5 bits/param effective against km4's ~4.03 — about 3 GB of the bundle.

The down projection's K of 704 is padded to 768 by the kernel's automatic _kpad256 treatment — 64 zero columns, ~0.5% wasted expert bytes. gate/up need no padding.

The gather kernel is load-bearing. Without MetalSwitchGLU, the sparse branch lowers to a dense matmul that reads all 128 experts every token — a 16× over-read.

New source modules

Three files were authored for this port; nothing in the shared toolchain was modified:

File	Contents
coreai_models/models/macos/gemma4_moe_text.py	Gemma4MoeConfig, Gemma4MoeRouter, GeluGLU, Gemma4MoeDecoderLayer, Gemma4MoeForCausalLM, and a loader that splits the checkpoint's fused gate_up_proj [E, 2*704, 2816] into the SwitchGLU's separate stacks
coreai_models/models/macos/gemma4_moe_pipelined.py	Gemma4MoePipelinedForCausalLM — subclasses the dense pipelined core and overrides one method; the attention half is inherited unchanged
coreai-model-zoo/conversion/export_gemma4_moe_decode_pipelined.py	the recipe

Contents

Path	Bytes	Files	Manifest context	Functions
gpu-pipelined/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8	17,580,059,414	8	16384	main (decode only)

File	Bytes
…aimodel/main.mlirb	17,547,866,346
…aimodel/main.hash	32
…aimodel/metadata.json	105
metadata.json (bundle manifest)	689
tokenizer/tokenizer.json	32,169,626
tokenizer/chat_template.jinja	18,683
tokenizer/tokenizer_config.json	3,730
tokenizer/generation_config.json	203

Manifest context is 16384. --max-ctx sets language.max_context_length and nothing else; lower it if the machine cannot afford the KV — that is a metadata edit, not a re-export.

Stop token: eos_token = "<turn|>" (id 106), the turn terminator Gemma 4 emits, applied by the export script itself. A host that stops on the raw upstream <eos> will overrun every reply.

Decode only, and a prefill variant is not deliverable by a flag

There is no prefill function. Two independent kernel blockers:

1. --metal-sdpa's flash-decode kernel is structurally q=1 with no causal mask. The full attention layers need that kernel for the same scratch-heap reason the dense 12B/31B do.
2. MetalSwitchGLU.forward is itself decode-only — it asserts a token batch of 1 and expands the single activation row across the routed slots. A BatchedMetalSwitchGLU (sort-by-expert grouped GEMM) exists for q>1, so this half is reachable, but it has never been exported for this model.

A prefill variant therefore needs a new flash-prefill Metal kernel plus the first export of the batched MoE kernel, plus a numerics gate for each. Prompts are prefilled one token at a time.

Requirements

• Apple silicon Mac with ≥24–32 GB unified memory, Core AI runtime.

• Engine contract: 2 inputs (input_ids, position_ids) → logits, one growing KV pair, no static inputs and no per-step mask.

• States: keyCache / valueCache Float16, 30 × 1 × 8 × ? × 512 — one growing pair, 30 slots (no KV sharing). The dynamic sequence dim means the runtime resolves GrowingKVCache, not a static allocation at the manifest maximum.

• KV cost: 491,520 bytes per token of context (fp16):

  Context	KV
  4096	2.01 GB
  8192	4.03 GB
  16384	8.05 GB
  32768	16.1 GB

• Resident weights, ~17.6 GB:

  	bytes
  Routed experts, aff4 (incl. _kpad256 zero columns)	~14.7 GB
  Attention + dense MLP + untied head, int4 block-32	~1.4 GB
  embed_tokens, fp16 in-graph	~1.5 GB
  Router, fp16	~0.02 GB
  Total	~17.6 GB
  + KV at 4096	~19.6 GB
  + KV at 16384	~25.7 GB

  Tier	Verdict
  16 GB	no — weights alone are ~1.6× the entire ~10.7 GB Metal working set
  24–32 GB	yes at ctx ≤ 8192 (~21.6 GB); 16384 is tight
  64 GB	yes, at any context

  This table is arithmetic from state shapes and weight formats — a prediction to check, not a measured tier claim. The residency mechanism for MoE bundles on macOS (mmap'd vs wired expert slabs) has not been measured.

  25 of the 30 layers have head_dim 256 zero-padded to 512, and all 30 slots grow linearly even though 25 of them only ever attend a 1024-token window. A ring-buffered sliding cache would be roughly 5× cheaper per token; that is model authoring, not a flag.

• The bundle manifest declares runtime_env COREAI_CHUNK_THRESHOLD=1.

Performance

No measurement exists, on any machine. Every figure in this card is either a byte count, an export-host measurement, or derived from state shapes and weight formats. No token has been generated from this bundle.

The intended argument for this model is throughput at comparable size against the dense 31B — 17.6 GB vs 20 GB, with ~4B active parameters per token read through a gather kernel — and quality per token against the 12B at ~2.1× the footprint. The nearest measured analogue is LFM2.5-8B-A1B at 140 tok/s through its gather kernel versus 39 tok/s for the same weights read as a dense over-read. Neither claim is measured for this bundle.

Status

Artifact	Status
gpu-pipelined/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8	UNQUALIFIED — never executed on target hardware. Established: the checkpoint is the QAT source, the graph exports, the producer fingerprint passes, and the MoE block wiring matches the reference implementation to 3.5e-7 relative error on real layer-0 weights (below).

A qualifying run would need, in priority order:

1. A decode oracle — generate and compare against an fp32 HF reference. This is the gate that catches an int4 or kernel-level error which the authoring gate below cannot see.
2. Isolate the two Metal kernels. gather_qmm (the aff4 variant) and the flash-decode SDPA have never been in the same graph, on any model. If decode produces garbage, re-export without --metal-sdpa first — it is an optimisation, the MoE kernel is not.
3. Measure residency, both dirty footprint and maximum resident including mmap. No MoE bundle has been measured this way.
4. Compare tok/s against the dense 12B and 31B on the same machine.
5. Router sanity at depth. 128 experts / top-8 with a learned per_expert_scale is far sparser routing than the LFM2.5-8B-A1B's 32/top-4. Check expert-utilisation spread on a real prompt set before trusting long-form output.

Verification

MoE block wiring verified against the reference implementation. transformers 4.57.6 (the pinned export environment) has no gemma4, so the gate transcribes Gemma4TextDecoderLayer.forward (the MoE-block half), Gemma4TextRouter.forward and Gemma4TextExperts.forward from upstream and runs both implementations on the real layer-0 weights in fp32:

T=1: max|dFFN|=7.629e-05  rel=3.453e-07  |  same top-8 ids: True  max|dw|=0.000e+00
T=4: max|dFFN|=9.537e-05  rel=2.735e-07  |  same top-8 ids: True  max|dw|=0.000e+00

Identical expert selection, bit-identical routing weights, ~3.5e-7 relative error on the block output. This gates the authoring — the wiring, the fused-weight split, the activation choice, the norm placement, the router maths. It does not gate the exported graph, the int4 rounding, or either Metal kernel. The attention half was not re-gated: it is the dense Attention class already used in the 12B and 31B bundles.

License

The upstream QAT checkpoint carries Apache-2.0 metadata and is ungated, but it is a Gemma model and the Gemma terms apply downstream, which is why this repo declares license: gemma. Use of these weights is subject to the Gemma Terms of Use, the Gemma Prohibited Use Policy and the Gemma 4 license. Those obligations travel with any redistribution of this bundle. The contribution here is the port and the recipe, not the weights.

hybrid-mf64/ bundles

EXPERIMENTAL — not qualified. These bundles have not passed a Mac-side oracle/parity gate or a device benchmark. Do not route production traffic to them until they have.

What this is. The 26B-A4B with a prefill entrypoint, from one copy of the weights the published gpu-pipelined/ decode bundle already ships. Two things had to be per-entrypoint at once, and both are:

• Experts. BatchedMetalSwitchGLU sorts the S*k (token, expert) pairs of a 64-token chunk by expert id so each expert slab is read once, and falls through to MetalSwitchGLU.forward verbatim whenever b*s == 1. One metalization, both entrypoints, decode arithmetic unchanged — main traces the identical q=1 gather matvec the published decode bundle carries.
• Attention. The 26B inherits the dense 12B/31B full-attention layers, so it needs the flash-decode Metal kernel for the scratch-heap reason, and that kernel is structurally q=1. SplitSDPA branches on query.shape[2], which is a concrete int in each trace, so main gets the kernel and prefill gets the MPSGraph composite (query axis + is_causal=True intra-chunk mask). Same technique as the 31B hybrid-pf64/ bundle.

Both kernels are registered on one converter before either add_pytorch_module.

This is mf64, not mf64-tp, and it cannot be -tp today. The token-tiled prefill kernel (moe_metal_tiled, the 8B-A1B mf64-tp experiment) is sym8-only — TiledBatchedMetalSwitchGLU.__init__ raises for any other scheme. The 26B ships aff4 experts, and that is not incidental: the checkpoint is QAT-trained on the ggml q4_0 per-block-32 grid, which an affine int4 block-32 grid represents exactly. An aff4 tiled kernel is future work, gated on two things — the 8B mf64-tp Mac verdict proving the tiling principle at all, and then an aff4 equality proof of the kind moek_tilegate.py gives the sym8 kernel. What this bundle uses instead is the shipped batched path, the one behind the 8B mf64 bundle that has a Mac measurement.

Measured on the conversion box (Linux x86_64, no GPU). Export wall 305 s; peak RSS 87.5 GB (well under the 12B's 68.7 → 31B's 148.9 band this was budgeted against, and never near the 75% kill line); bundle 17,580,916,612 B; producer coreai-core 1.0.0b2; ctx 16384; function_map {"main": ["main", "prefill"]}. Trace accounting: all 5 full layers took the metal branch at S=1 and the composite branch at S=64, 5/5 both ways — the export fails if either count is 0. 30 MoE layers metalized. Graph bytecode declares input_ids, position_ids, keyCache, valueCache, logits, prefill, 90 gather_qmm_int4aff call sites (30 layers x gate/up/down) and gemma4_dense_full_sdpa_occ8 — 2 inputs, no mask, which is the arity CoreAISequentialEngine requires.

Numerics gate (box-side, gate_hybrid_sdpa.json). The 26B is a third distinct block-GQA mapping and was gated as its own case: 16 query heads, 2 global KV heads replicated across 8 cache slots, so the kernel's kv = h / (H / slots) = h//2 must land on real head (h//2)//4 = h//8. Kernel reference vs the composite at those exact shapes: max abs 2.50e-2 (S=137) and 1.17e-2 (S=1024) — the composite's own fp16 error. Against an fp32 ground truth the kernel is the more accurate of the two: 9.74e-4 / 74.9 dB at S=137 and 9.76e-4 / 75.1 dB at S=1024, versus the composite's 53.0 and 57.9 dB. The 12B (1 global head, rep 8) and 31B (4 global heads, rep 4 over 16 slots) rows are in the same file as controls; a mapping bug that cancelled in those would show here. MetalDenseFullSDPA also evaluates in eager at 26B dims and matches its own reference exactly (max abs 0.0). This gates the mapping and the scale, not the Metal source (no GPU on the box).

Tier — read the arithmetic, not the bundle size. KV is 2 x 30 layers x 8 KV slots x 512 head_dim x fp16 = 491,520 B/token:

Context	KV	+ 17.6 GB weights	Verdict
4096	2.01 GB	19.6 GB	32 GB machine
8192	4.03 GB	21.6 GB	32 GB machine, marginal against a ~21-24 GB working set
16384 (this manifest)	8.05 GB	25.6 GB	64 GB machine

So: a 32 GB Mac can run it at shallow context, the manifest's full 16384 wants 64 GB, and no 16 GB machine runs it at any context.

EXPERIMENTAL — unqualified until a big-machine measurement exists. Nothing here has produced a token. In priority order:

1. Does it load and decode? Decode should be indistinguishable from the published gpu-pipelined/ bundle: same weights, same q=1 gather kernel, same flash-decode SDPA. A decode regression against that bundle would mean the batched wrapper is not falling through at b*s == 1 as its source says it does.
2. Prefill throughput. The falsifiable prediction: the batched kernel's anchor is the 8B-A1B mf64 bundle's measured 6.3 ms/prompt-token, and this bundle should land in that mechanism's class rather than the 21.5 ms/token of a decode-only bundle walking the prompt one token at a time. Do not read 6.3 ms as the expected number for this model: the 26B activates ~4B parameters per token against the 8B-A1B's ~1B, and its dense MLP runs in parallel with the expert branch, so a figure several times 6.3 ms would still confirm the mechanism. What would falsify it is prefill landing at or above the per-token cost of decode — that would mean the sorted grouped-GEMM path is not engaging and the chunk is being walked serially.
3. Chunk parity. A prompt walked in 64-token chunks must give the same logits as the same prompt walked token by token. Two independent reasons to check it here rather than assume: the entrypoints use different attention implementations, and the expert path does a sort/unsort round trip that the q=1 path does not.
4. Compare against the dense 31B on the same machine. That is the whole case for this model — 17.6 GB and ~4B active per token against 20 GB and 31B dense.

Nothing already published was touched. gpu-pipelined/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8 is unchanged and remains the only 26B artifact anyone has run.

Bundle	Source checkpoint	Context	Functions	Bytes	Producer	Created
hybrid-mf64/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8_pf64	google/gemma-4-26B-A4B-it-qat-q4_0-unquantized	16384	main+prefill	17,580,916,612	coreai-core 1.0.0b2	20260818T141434Z