Kimi-K3-GGUF

GGUF quantizations of moonshotai/Kimi-K3, mirrored at AtomicChat/Kimi-K3.

Kimi K3 is a ~2.9T parameter (104B active) mixture of experts model: 93 layers, 896 experts with 16 active per token, 7168 hidden size, 1M context, trained natively in MXFP4. The original only ships in MXFP4, so everything here comes from a direct MXFP4 to Q8_0 conversion, and the lower quants are requantized from that Q8_0.

Honesty first: this thing is enormous

Let's be upfront. This is a monster. The smallest quant we make, IQ1_S at ~1.56 bits, still weighs about 540 GB. That already blows past the 512 GB that even well specced machines top out at. To actually run any of these you need enough fast memory (RAM plus VRAM combined) to hold the whole file. In practice that means a large RAM server or a serious multi GPU rig, not a laptop and not most workstations.

We know most people won't be able to run these locally, and that's okay. We made them for the few setups that can, for people who want to poke at the model, and as an archival base so the quants already exist the day the hardware or the tooling catches up. Rule of thumb: grab the biggest quant your memory can actually hold.

Does it even work?

Yes. While computing the importance matrix we ran a full perplexity pass over calibration_datav3, and Q3_K_S came out at PPL 3.85 (ctx 512). That is a healthy number. It means the quantized model is coherent and not broken (a broken quant gives perplexity in the tens or hundreds, or NaN), and expert coverage in the imatrix landed at 99.4 to 99.8 percent.

Fair warning: that figure is measured on the calibration data, so read it as a sanity signal, not a leaderboard score. Once the quants finish we'll run proper held out perplexity (wikitext) across all of them and drop the table right here.

Requirements

kimi_k3 is not in mainline llama.cpp yet, so you build our fork that adds support. It lives in this PR: AtomicBot-ai/atomic-llama-cpp-turboquant#58.

git clone https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant
cd atomic-llama-cpp-turboquant
git fetch origin pull/58/head:kimi-k3 && git checkout kimi-k3
cmake -B build -DGGML_CUDA=OFF -DLLAMA_CURL=OFF
cmake --build build -j --target llama-cli

Then point llama-cli at the first shard of any quant (the rest load on their own):

./build/bin/llama-cli -m ./IQ2_XXS/Kimi-K3-IQ2_XXS-00001-of-000NN.gguf -p "Hello"

Plain mainline llama.cpp will reject these files with an unknown architecture error until Kimi K3 support lands upstream (tracking: ggml-org/llama.cpp#26185).

Text only for now. Kimi K3 is multimodal, but these GGUFs are the language model on its own, so they run as text to text and image input does not work yet. We are actively working on vision: it needs both an mmproj file (the MoonViT vision tower converted to GGUF) and MoonViT support in llama.cpp's clip/mtmd code. Both are in progress. When it lands, one small mmproj will pair with any of the LLM quants above, so nothing here has to be re-downloaded.

Quants

Each quant is a folder of 45 GB shards. Point llama.cpp at the first shard (...-00001-of-000NN.gguf) and the rest load on their own. Sizes for quants that aren't up yet are estimates.

Folder	Bits	Size	imatrix	Status	Notes
IQ1_S	1.60	~555 GB	yes	uploaded	smallest, extreme, quality degraded
IQ1_M	1.75	~610 GB	yes	planned
IQ2_XXS	2.06	~715 GB	yes	planned	best size to quality in the 2 bit tier
Q2_K	2.9	~1009 GB	no	uploaded
Q3_K_S	3.4	~1196 GB	no	uploading
Q8_0	8.5	~2955 GB	no	uploaded	conversion base, for archival and requant, not for running
imatrix.dat	n/a	small	n/a	uploaded	importance matrix, see below

We deliberately skip the middle i-quants (IQ2_XS, IQ2_S, IQ3, IQ4). At roughly 0.8 to 1.2 TB they land right on top of Q2_K and Q3_K_S, which already cover that range and are simpler. The set above spans the useful spectrum without pointless duplicates.

Measured quality (perplexity)

Perplexity on the wikitext-2 test set (ctx 512, 50 chunks). Lower is better. Read these only relative to each other: absolute PPL is only comparable within the same model, dataset and context. We fill this in as each quant finishes.

quant	bits	PPL (wikitext-2, ctx 512)
IQ1_S	1.60	14.81
IQ1_M	1.75	pending
IQ2_XXS	2.06	pending
Q2_K	2.9	pending
Q3_K_S	3.4	pending

For reference, during the imatrix pass Q3_K_S measured PPL ~3.85 on the calibration corpus, so expect the full table to run from roughly 15 (IQ1_S, heavy 1.6-bit compression, degraded but coherent) down toward ~4 at Q3_K_S. A held-out KL-divergence table (versus Q3_K_S as the base) will follow.

About the imatrix

imatrix.dat was computed with the calibration_datav3 corpus and is applied to every i-quant (IQ1 and IQ2).

Being straight about how it was made:

• The source is the Q3_K_S quant, not the full Q8_0. Kimi K3 multi GPU inference doesn't work in the fork yet, and the full 3 TB Q8 doesn't fit in RAM for CPU inference, whereas Q3_K_S (about 1.2 TB) fits and runs cleanly on CPU. Computing an imatrix from a quantized model is standard practice, and its effect on the final 1 to 2 bit outputs is second order next to the quantization itself.
• With 896 experts and only 16 firing per token, a handful of rarely used experts stay slightly undersampled and fall back to default quantization for those weights.

Usage

# download one quant folder
hf download AtomicChat/Kimi-K3-GGUF --include "IQ2_XXS/*" --local-dir ./kimi-k3

# run with the fork build
./llama-cli -m ./kimi-k3/IQ2_XXS/Kimi-K3-IQ2_XXS-00001-of-000NN.gguf -p "Hello"

Attribution and license

This is a derivative of Kimi K3 by Moonshot AI, redistributed under the Kimi K3 License. The original copyright notice and license are kept in LICENSE.

Kimi K3 License
Copyright (c) 2026 Moonshot AI

Quantized and published by AtomicChat.