Instructions to use batiai/GLM-5.2-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use batiai/GLM-5.2-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="batiai/GLM-5.2-GGUF",
	filename="zai-org-GLM-5.2-IQ2_XS-00001-of-00005.gguf",
)

output = llm(
	"Once upon a time,",
	max_tokens=512,
	echo=True
)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use batiai/GLM-5.2-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf batiai/GLM-5.2-GGUF:IQ2_XS
# Run inference directly in the terminal:
llama cli -hf batiai/GLM-5.2-GGUF:IQ2_XS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf batiai/GLM-5.2-GGUF:IQ2_XS
# Run inference directly in the terminal:
llama cli -hf batiai/GLM-5.2-GGUF:IQ2_XS

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf batiai/GLM-5.2-GGUF:IQ2_XS
# Run inference directly in the terminal:
./llama-cli -hf batiai/GLM-5.2-GGUF:IQ2_XS

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf batiai/GLM-5.2-GGUF:IQ2_XS
# Run inference directly in the terminal:
./build/bin/llama-cli -hf batiai/GLM-5.2-GGUF:IQ2_XS

Use Docker

docker model run hf.co/batiai/GLM-5.2-GGUF:IQ2_XS

LM Studio
Jan

vLLM

How to use batiai/GLM-5.2-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "batiai/GLM-5.2-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "batiai/GLM-5.2-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/batiai/GLM-5.2-GGUF:IQ2_XS

Ollama
How to use batiai/GLM-5.2-GGUF with Ollama:
```
ollama run hf.co/batiai/GLM-5.2-GGUF:IQ2_XS
```

Unsloth Studio

How to use batiai/GLM-5.2-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for batiai/GLM-5.2-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for batiai/GLM-5.2-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for batiai/GLM-5.2-GGUF to start chatting

Atomic Chat new
Docker Model Runner
How to use batiai/GLM-5.2-GGUF with Docker Model Runner:
```
docker model run hf.co/batiai/GLM-5.2-GGUF:IQ2_XS
```

Lemonade

How to use batiai/GLM-5.2-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull batiai/GLM-5.2-GGUF:IQ2_XS

Run and chat with the model

lemonade run user.GLM-5.2-GGUF-IQ2_XS

List all available models

lemonade list

GLM-5.2 GGUF — Quantized by BatiAI

IQ2_XS / IQ3_XXS / Q3_K_M quantization of zai-org/GLM-5.2 (744B total / 40B active MoE). Quantized directly from official Z.AI weights by BatiAI.

Why GLM-5.2?

744B parameters (40B active) — frontier MoE with Deep Sparse Attention (DSA)
Latest in Z.AI's GLM-5 series — successor to GLM-5.1, the #1 open-source model on SWE-Bench Pro
256 routed experts + 1 shared expert (top-8 routing + DSA indexer) — extreme sparsity
78 transformer blocks (3 dense + 75 MoE) with hybrid attention/FFN routing
1M token context (max_position_embeddings=1048576, RoPE θ=8M)
MIT license — fully permissive for commercial use, fine-tuning, redistribution
Released by Z.AI / Zhipu AI — same lineage as ChatGLM / GLM-4 / GLM-5.1

Quick Start

# IQ2_XS (smallest — 214 GiB, dynamic 2.44bpw, fits Mac Studio 256GB)
hf download batiai/GLM-5.2-GGUF --include "*IQ2_XS*"

# IQ3_XXS (273 GiB — needs Mac 384GB+ / server)
hf download batiai/GLM-5.2-GGUF --include "*IQ3_XXS*"

# Q3_K_M (highest-fidelity offered — 337 GiB, needs 384GB+ unified RAM)
hf download batiai/GLM-5.2-GGUF --include "*Q3_K_M*"

Available Quantizations

Quant	Total Size	Shards	Min RAM	Target Hardware
IQ2_XS ⭐	230 GB / 214 GiB (measured)	5 @ 48 GB max	~256 GB	Mac Studio 256GB / server
IQ3_XXS	293 GB / 273 GiB (measured)	7 @ 48 GB max	~320 GB	M3 Ultra 384–512GB / H100 node
Q3_K_M	361 GB / 337 GiB (measured)	8 @ 48 GB max	~384 GB	M3 Ultra 512GB / H100 node

IQ2_XS uses a dynamic recipe — routed experts at IQ2_XS (2.31 bpw) while attention / router / shared-expert / DSA indexer / MTP block are protected at Q4–Q6 → effective 2.44 bpw. This opens the 256GB Mac Studio tier that IQ3_XXS / Q3_K_M (512GB-class) cannot reach. Q3_K_M is also the imatrix-source build (kept in RAM cache during calibration). All sizes above are measured. IQ4_XS (~376 GB, 512GB-class) overlaps IQ3/Q3's hardware tier, so it is archived on NAS and re-uploadable on request rather than shipped by default.

⚠️ High-RAM only — the smallest build (IQ2_XS, 214 GiB) needs Mac Studio 256GB+; IQ3_XXS / Q3_K_M need 384–512GB. It does not fit a 192GB Mac. 16–192GB Mac users should use batiai/qwen3.6-35b or batiai/minimax-m2.7 instead.

Hardware Reality Check

Your System	IQ2_XS (214 GiB)	IQ3_XXS (273 GiB)	Q3_K_M (337 GiB)
Mac ≤192GB	❌ Won't fit	❌	❌
Mac 256GB	✅ Usable (~30GB headroom)	❌ Won't fit	❌
Mac 384GB	✅ Comfortable	✅ Usable	✅ Tight
Mac M3 Ultra 512GB	✅ Comfortable	✅ Comfortable	✅ Usable
2× M3 Ultra (cluster)	✅ Fast	✅ Fast	✅ Fast
8× A100 80GB (640GB)	✅ Fast	✅ Fast	✅ Fast
H100 node	✅ Fast	✅ Fast	✅ Fast

Numbers based on MoE activation pattern — 40B active params × 2 bytes (Q4 active) ≈ 80GB runtime, plus shard buffers + KV cache. Going below the min RAM forces SSD paging which destroys throughput.

Special Engineering Notes

GLM-5.2 uses Deep Sparse Attention (DSA) — a per-layer "indexer" tensor selects the top-K key positions for sparse attention. The config marks 21 of 78 layers as full indexer (layers 0–2 plus every 4th from layer 6: 6, 10, 14, … 74); the remaining 57 are shared. This hybrid layout required two fixes during quantization:

DSA indexer tensors not in imatrix — --tensor-type indexer=q5_k override (these tensors are not exercised by the forward pass used for imatrix, so they have no calibration data and are kept at Q5_K)
MTP / NextN block (blk.78) imatrix gap — GLM-5.2 ships num_nextn_predict_layers=1, so the multi-token-prediction block lands at blk.78 (= num_hidden_layers, derived dynamically by the pipeline as LAST_BLK=NLAYERS=78). It is not used in the forward pass and has no imatrix data; --tensor-type blk.78=q5_k keeps it at higher precision.

Both flags are baked into our quantization pipeline (scripts/runtime/glm52-pipeline.sh). The fallback Q5_K layers add a negligible amount to file size but prevent low-bit IQ-quants from bailing on missing imatrix data. The entire pipeline runs on NAS (/mnt/nas, 1.51 TB) under bati.cpp.

What BatiAI's Quantization Delivers

	BatiAI	typical 3rd-party
Source	Direct from official Z.AI weights	Often re-quantized from other GGUFs
Quantization flow	safetensors → Q8_0 → Q3_K_M / IQ3_XXS / IQ4_XS with imatrix (wikitext-2-raw, 200 chunks)	Varies
imatrix	✅ 200 chunks (quality saturation), measured over Q3_K_M in RAM cache	Often skipped or fewer chunks
DSA indexer handling	✅ Q5_K override documented	Often unaddressed → garbage low-bit
MTP / last-block imatrix gap	✅ Workaround applied (blk.78=q5_k)	Often causes bail-out or quality loss
BatiAI signature	✅ `general.author=BatiAI`, `general.url=https://flow.bati.ai`	✗

Model Comparison — BatiAI Lineup

Your Hardware	Best BatiAI Model	Size
16GB Mac	`batiai/gemma4-e4b:q4`	5GB
24GB Mac	`batiai/gemma4-26b:iq4`	15GB
48GB Mac	`batiai/qwen3.6-35b:iq4`	22GB
96GB Mac	`batiai/qwen3.6-35b:q6`	29GB
128GB Mac	`batiai/minimax-m2.7:iq3`	82GB
192GB Mac Studio	`batiai/kimi-k2.6:iq3`	394GB (paged)
M3 Ultra 512GB	GLM-5.2 IQ4_XS ⬅ here	~376GB
M3 Ultra 512GB (alt)	`batiai/kimi-k2.6:iq4`	546GB (heavy swap)

GLM-5.2 IQ4_XS at ~376 GB is among the largest models that run on a single M3 Ultra 512GB without crippling SSD swap. Kimi K2.6 IQ4 (546GB) would page heavily on the same machine.

Benchmarks (source model)

Benchmark numbers pending. No locally-verified scores yet — values will be populated by scripts/render-readmes.py from bench.sh results on target hardware. GLM-5.2 is the successor to GLM-5.1 (the #1 open-source model on SWE-Bench Pro); quantization-preserved scores are not asserted until measured.

Validating that quantization preserves the source model's quality on Mac M3 Ultra is pending (bench.sh on target hardware). No unverified scores are listed here.

Technical Details

Original Model: zai-org/GLM-5.2
Architecture: GlmMoeDsaForCausalLM (model_type=glm_moe_dsa) — 744B total / 40B active, 78 blocks (3 dense + 75 MoE), 256 routed + 1 shared expert (top-8, noaux_tc routing, sigmoid scoring, routed_scaling_factor=2.5), DSA hybrid attention with ~21 full-indexer layers
Dimensions: hidden 6144, intermediate 12288, MoE intermediate 2048, 64 attn heads / 64 KV heads, head_dim 192 (qk 256 = 192 nope + 64 rope, v 256), kv_lora_rank 512, q_lora_rank 2048, vocab 154880, dtype bfloat16
Context: 1M tokens (max_position_embeddings=1048576, rope_theta=8000000, interleaved RoPE)
MTP: num_nextn_predict_layers=1 → NextN block at blk.78
Original storage: BF16/FP8 mix (~1.4 TB safetensors)
License: MIT
Quantized with: bati.cpp (BatiAI's llama.cpp fork — needed for the glm-dsa / DSA architecture; hybrid-DSA loader treats the indexer as optional per layer, blk.78 in-place metadata patch)
Calibration: wikitext-2-raw, 200 chunks (quality saturation), imatrix measured over Q3_K_M
imatrix overrides: --tensor-type indexer=q5_k --tensor-type blk.78=q5_k
Quantized by: BatiAI

Usage

llama.cpp / bati.cpp

GLM-5.2 currently requires bati.cpp (BatiAI's llama.cpp fork) — mainline ggml-org/llama.cpp does not yet support the glm-dsa (GLM-MoE-DSA) DSA indexer at runtime. Will switch to mainline once full DSA-indexer support lands.

git clone https://github.com/batiai/bati.cpp.git
cd bati.cpp
cmake -B build -DGGML_METAL=ON   # macOS
# or: cmake -B build -DGGML_CUDA=ON   # Linux
cmake --build build -j --target llama-cli

hf download batiai/GLM-5.2-GGUF --include "*IQ4_XS*" --local-dir ./glm52

# Point at shard 00001 of the IQ4_XS split set (filename pattern below)
build/bin/llama-cli -m ./glm52/zai-org-GLM-5.2-IQ4_XS-00001-of-*.gguf \
    -p "Your prompt" \
    --ctx-size 32768 \
    --n-gpu-layers 99

Ollama

Not available on Ollama. Ollama vendors mainline llama.cpp, which does not support the glm-dsa (GLM-MoE-DSA) DSA indexer at runtime, so these GGUFs cannot load there. Mainline PR #19460 adds the GLM-MoE-DSA architecture but the DSA indexer is a follow-up PR still pending; as of 2026-06-20 ollama.com/batiai/GLM-5.2 returns 404. Quantization and inference are possible only with batiai/bati.cpp. An Ollama build will be revisited once the mainline glm-dsa indexer is merged.

vLLM / TGI

Not directly compatible — these serve FP8/BF16 safetensors. Use original zai-org/GLM-5.2 for vLLM.

About `bati.cpp`

batiai/bati.cpp is BatiAI's llama.cpp-based fork focused on:

Apple Silicon (Metal) optimization
Frontier-model early access (V4-Flash, GLM-5.1 / GLM-5.2 DSA, etc.) before mainline merges
BatiAI quantization standard (signature, imatrix workflow)

Built on top of ggml-org/llama.cpp and antirez/llama.cpp-deepseek-v4-flash (all MIT). See bati.cpp/ATTRIBUTION.md for full credits.

License

Inherits the source model license: MIT. Public (non-gated) — MIT permits commercial use, fine-tuning, and redistribution with no access form required.

About BatiFlow

BatiFlow — free on-device AI automation for Mac. 5MB native app, 60+ tools (KakaoTalk, iMessage, Slack, Calendar, Notes, Chrome, file system). Works with all batiai/* models.

Downloads last month: 520

GGUF

Model size

753B params

Architecture

glm-dsa

Hardware compatibility

2-bit

3-bit

Model tree for batiai/GLM-5.2-GGUF

Base model

zai-org/GLM-5.2

Quantized

(94)

this model