Qwen3.5-0.8B — LiteRT-LM

Update 2026-08-13: re-converted from the same weights with the gated-delta-rule chunk kernel re-expressed in an all-rank-≤-4, pad-free form the GPU delegate executes correctly. The graph now delegates fully to the GPU and answers correctly there — verified end-to-end on macOS and iPhone 17 Pro (Metal), where GPU decode runs ~2.8× the CPU. CPU behaviour and speed are unchanged. GPU execution requires fp32 activations, which the bundle now declares; see the honest notes for the memory cost. (While rewriting we found the GPU delegate miscomputing one op shape — reported upstream as LiteRT#9272; the kernel now avoids that shape.)

Update 2026-08-07: metadata-only fix (weights unchanged) — <|im_end|> is now declared as a stop token alongside <|endoftext|>. Qwen3.5 uses different tokens for the chat turn-end and config.json's eos_token_id; with only the latter declared, the literal <|im_end|> text leaked into litert-lm run output (generation still stopped correctly). Replies are now clean, and multi-turn history no longer records the marker as text.

Qwen/Qwen3.5-0.8B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 (both backends gated on 0.15.0 and 0.16.0). To our knowledge this is the first Qwen3.5 in LiteRT form, and the first gated-delta-net hybrid served by the released LiteRT-LM runtime — and, since the 2026-08-13 update, the first to run its delta-rule scan on a mobile GPU.

Qwen3.5 is Alibaba's hybrid architecture: GatedDeltaNet (gated delta rule linear attention) blocks interleaved with a few gated full-attention blocks (this 0.8B model has 18 linear-attention + 6 attention layers). The linear-attention blocks carry constant-size per-layer conv + recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 6 attention layers keep KV (4096-token budget here). The upstream 0.8B checkpoint is multimodal; this package is the text decoder only (the vision tower and MTP heads are dropped exactly as upstream's own Qwen3_5ForCausalLM text-only load contract does).

File	Recipe	Size
Qwen3.5-0.8B_int8.litertlm	int8 dynamic on linears + embedding (convs and the delta rule stay float)	963 MB

Correctness

All gates below were run on this exact file (or its float parent) after the 2026-08-13 rewrite:

• Logits-level parity vs PyTorch: the float export matches the HF model teacher-forced across 48 positions — top-1 and top-5 agreement 100%, Pearson 1.0000, KL ≈ 0.
• 8-question sanity gate: 8/8 on CPU and GPU, on both litert-lm 0.15.0 and 0.16.0 (Mac). On iPhone 17 Pro the GPU run answers 8/8.
• Prompt-length robustness: first-token check against the runtime's real prefill chunk plans, fresh engine per length — 40/40 lengths clean on CPU, 20/20 on GPU.
• Multi-turn conversations carry state correctly across turns.

Quality — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, max-tokens 512, n=100, same harness, prompt and answer extraction for both rows (non-thinking mode, matching the bundled template):

Configuration	GSM8K
PyTorch bf16 (reference, MPS)	12%
LiteRT int8 (this file)	11%

The absolute level is the 0.8B non-thinking model's own (thinking disabled, 512-token budget); the one-point gap is what the conversion + int8 cost.

Usage

litert-lm run ./Qwen3.5-0.8B_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./Qwen3.5-0.8B_int8.litertlm --backend gpu --cache no --prompt "..."

Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks.

Chat template note: the bundle ships a simplified ChatML template rather than the stock Qwen3.5 template. Thinking is disabled the way the stock template's non-thinking mode does it (an empty <think>\n\n</think> block opens each assistant turn), and — deliberately — that block is kept in history renders too: the stock template strips it from past turns, which breaks LiteRT-LM's incremental conversation rendering (the engine requires each turn's render to be a string-extension of the previous one) and kills multi-turn on turn 2. Tool-calling and vision sections are not included.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

Backend	Prefill (256)	Decode	TTFT
GPU	1972 tok/s	161.8 tok/s	0.14 s
CPU	666 tok/s	46.7 tok/s	0.41 s

CPU figures are essentially unchanged from the previous file (the delta-rule kernel was already matmul-form on CPU); the 2026-08-13 update adds the working GPU path.

On device (cold start, single runs, 138-token prompt, quality-gate harness):

Device	Backend	Prefill	Decode	TTFT	Peak memory
iPhone 17 Pro	GPU (Metal)	387 tok/s	41.4 tok/s	0.47 s	5.48 GB
iPhone 17 Pro	CPU	170 tok/s	14.6 tok/s	0.93 s	1.21 GB

Pixel 8a (Tensor G3, litert_lm_main built from the v0.16.0 tag, 260-token prompt with generation running long, 3 runs):

Backend	Prefill (260 tok)	Decode	TTFT
CPU	50–134 tok/s	8.0–13.6 tok/s	2.0–5.3 s
GPU (OpenCL)	— does not fit this phone's memory (see honest notes)

Honest notes:

• Where GPU execution is verified. macOS (Metal), iPhone 17 Pro (Metal) and Pixel 8a (Arm Mali, OpenCL). We have not verified GPU execution on Qualcomm Adreno devices — if you are on a Snapdragon phone, use the CPU backend unless you have confirmed the GPU path on your own device.
• GPU inference runs with fp32 activations (declared in the bundle; an fp16-activation formulation is unfinished — the residual issue is a real-weight fp16 range overflow in one layer-0 head, a property of the checkpoint, not of the conversion). That is where the GPU memory multiple comes from — 5.48 GB vs 1.21 GB on iPhone above.
• Pixel 8a cannot compile this file on its GPU: fp32-expanded weights plus the full prefill-ladder of compiled programs exceed the phone's ~3.8 GB available memory (a reduced dev build of the same graph runs and answers correctly there, fully delegated — the limit is memory, not ops). CPU works as before on Android.
• On low-end Android GPUs, decode is memory-bandwidth-bound and does not beat the CPU anyway; the GPU win is on Apple hardware (and, generally, prefill/TTFT).
• On quality: the per-question sanity gate is word-for-word identical to HF fp32, but on a harder composite probe (8 questions in one 138-token prompt) the int8 quantization measurably costs answers at this 0.8B scale. If you need maximum fidelity on complex prompts, ask for a float/fp16 variant.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm qwen35_work/):

• Rank-4 chunk kernel (the 2026-08-13 change): the reference chunked delta rule spells its contractions as broadcast-multiply-reduce over high-rank intermediates. The vendored kernel re-expresses them as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math), and writes every tail-pad as a concat with a zeros constant rather than a PAD op — the GPU delegate miscomputes rank-3 non-final-axis PAD (reported as LiteRT#9272). This is why the same weights got ~3.5× faster on CPU and became fully delegable on GPU.
• Export cache for GatedDeltaNet layers: conv [B, conv_dim, K] + recurrent [B, heads, k_dim, v_dim] cache layers registered for layer_types == "linear_attention", so torch.export traces the model's own state contract.
• State continuation tracing: prefill graphs trace the chunk-continuation branch (previous conv/recurrent state consumed, so multi-chunk prefill composes) and the decode graph traces the fused single-step branch (conv window rolled in place by causal_conv1d_update).
• Prefill-pad guard: the runtime's chunk planner runs partially-filled prefill chunks; pad positions are made identity steps for the delta rule (per-token decay forced to ~1, zeroed k/v injection) and the stored conv window is gathered at the last valid column. GPU trap: reductions in the guard keep their batch dimension (keepdim=True) — a rank-0 scalar entering broadcast arithmetic is silently miscomputed by the GPU delegate.
• Constant-eye chunk kernel: the reference kernel builds torch.eye inside the traced function, which lowers to a STABLEHLO_IOTA op no released TFLite kernel set registers; the identity matrix is lifted as a graph constant.
• Runtime state binding: litert-lm ≥ 0.15 binds per-layer states through an ExecutorMetadata section listing each state tensor; it is appended at package time.
• Quantization: post-hoc dynamic int8 over linears + embedding only; the convs and the delta rule stay float.
• GPU activation precision: the bundle's TOML declares prefer_activation_type = "fp32" — required for correct GPU numerics on this family today.

License and changes

Distributed under Apache-2.0 (inherited from the base model). Changes from the original work: text-decoder weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; vision tower and MTP weights omitted; tokenizer repackaged unmodified; chat template replaced with the simplified ChatML template described above. This repository is a community conversion and is not affiliated with Alibaba / the Qwen team.