Model Card for Qwen3.5-9B (4-bit GGUF, with vision)
A 4-bit GGUF build of Qwen3.5-9B packaged to run entirely on device, with the vision tower, the tokenizer, the merge table, the chat template and the sampling card all embedded in the single weight file. Nothing calls out to a server at inference time and there are no sibling files to keep in sync.
This repository holds the quantized weights only, not training data or the original checkpoint.
File format
.ggxf is GGUF plus the tensors and metadata keys this project adds, so these
files load in Gadeon and are not expected to load in other GGUF readers.
Files
| File | Purpose |
|---|---|
Qwen3.5-9B-UD-Q4_K_XL.ggxf |
everything: weights, vision tower, tokenizer, chat template, sampling card |
One file, nothing beside it. The vision tower travels inside the GGUF rather
than as a separate mmproj, and the sampling parameters travel with it, so a
runtime never has to guess them or borrow another model's settings.
Every block type here is a standard ggml one, inherited from the upstream quantization. The additions are the embedded vision tower and the metadata, and those have only been exercised in the companion on-device runtime -- third-party tools have not been tested against this file, and a tool that does not expect a tower inside the weight file may ignore it or refuse it.
Provenance
The language trunk and the multi-token-prediction drafter are unsloth's
UD-Q4_K_XL quantization with the MTP
block of Qwen3.5-9B,
used as published. What this build adds is the vision tower and the metadata: the
v.* / mm.* tensors and clip.* configuration, and the generation config
with its sampling matrix and its stop-token list. The tower is carried at
source precision (F16/F32).
Model Details
Model Description
Qwen3.5-9B is a dense hybrid. Its layers alternate three Gated DeltaNet linear-attention blocks to one full attention block. The linear blocks carry a small fixed-size recurrent state instead of a growing key/value cache, so memory stays flat as context grows, while the periodic attention blocks preserve exact long-range recall.
This build is multimodal. The vision tower is included, so image input works from this file alone.
This build carries the multi-token-prediction layer (blk.32, the
nextn block), so a runtime that speculates can draft from it; one that
does not decodes normally.
- Developed by: Qwen, Alibaba (base model); quantization by unsloth; packaging by leok7v
- Model type: Hybrid Gated DeltaNet + attention causal language model with a vision encoder
- Language(s): English and the languages of the base model
- License: Apache 2.0, inherited from the base model
- Quantized from model: Qwen/Qwen3.5-9B
Model Sources
- Repository (this build): https://huggingface.co/leok7v/Qwen3.5-9B
- Base model: Qwen/Qwen3.5-9B
- Trunk quantization: unsloth/Qwen3.5-9B-MTP-GGUF
Uses
Direct Use
On-device chat, summarization, drafting, question answering over supplied text, and description of supplied images, in offline or privacy-sensitive settings where prompts must not leave the device.
Out-of-Scope Use
Anything the base model's licence and card exclude. This build adds no alignment, no safety tuning and no filtering of its own; it is the upstream model in a different numeric format. Not a basis for legal, medical, financial or safety decisions, and not intended for high-stakes automated decisions without human review.
Bias, Risks, and Limitations
The base model's biases and failure modes carry over unchanged.
Four bits is a mild compression compared with the 1-bit and 2-bit builds in this family, but it is still lossy: outputs will not match the upstream model token for token, and the gap is widest where the model was already uncertain. Evaluate on your own task rather than assuming it behaves like the full-precision model.
Recommendations
Treat generations as drafts to verify, not as fact. For anything where accuracy matters, and for arithmetic in particular, check the result.
How to Get Started with the Model
The tokenizer, the chat template and the sampling card are embedded, so a runtime reads all of them from the GGUF. The chat template documents the roles, the thinking block and the tool-call format.
Generation stops on either <|im_end|> (248046) or <|endoftext|> (248044).
Both ids are carried in the embedded generation config and in
tokenizer.ggml.eos_token_ids; a runtime that reads only the scalar
tokenizer.ggml.eos_token_id will see <|im_end|> alone.
Technical Specifications
Model Architecture and Objective
| Property | Value |
|---|---|
| Objective | Autoregressive next-token prediction |
| Hidden size | 4096 |
| Layers | 32 (Gated DeltaNet + attention, three to one) |
| Attention heads | 16 query / 4 key-value |
| Head dimension | 256 (partial rotary, 64) |
| Linear-attention state | 128, 16 groups, inner size 4096 |
| Feed-forward | 12288 |
| Vocabulary | 248320 |
| Context length | up to 262144 tokens |
| Vision encoder | 27 blocks, 768px tiles, 2x2 spatial merge |
Weight format
6.57 GiB, 776 tensors (334 of them the vision tower at F16/F32), GGUF v3. The trunk is a MIXED-precision assignment rather than one block type throughout: the widths are chosen per tensor, so the most sensitive rows keep more bits than the bulk of the model.
Citation
Please cite the upstream Qwen3.5-9B model. See the base model card at Qwen/Qwen3.5-9B for the canonical reference and license.
Model Card Contact
leok7v on Hugging Face.