EschaLabs/escha-runtime-qwen3dense

Escha Runtime — qwen3dense

By Escha Labs Inc.

The serving runtime for Escha 2-/3-bit (escha) quantized models of the qwen3_5 dense architecture (Qwen3.8-27B and siblings). One repo per model architecture, one directory per engine — this architecture currently has one engine, sglang/.

	SGLang — sglang/
Best for	everything: single user, teams, agents
Concurrency	continuous batching, paged KV, optional radix prefix cache
Tool calls / JSON schema / thinking parser	yes
Interface	OpenAI-compatible (/v1/chat/completions, /v1/completions, /v1/models)
Install	Python 3.12 venv + CUDA-12 PyTorch, then one wheel

The engine is a fork of SGLang bundled inside the wheel, running the Escha CUDA kernels. No separate sglang install is needed, and none should be present — the wheel ships its own.

Compatible models

Model repo	Bits
EschaLabs/Qwen3.8-27B-Escha-W2	2-bit, mixed-rate (escha)

This runtime targets the qwen3_5 dense architecture. Its wheel also happens to register the eschamoe mixture-of-experts method, so a qwen3_5_moe model will load too — but the tuning, the defaults in sglang/serve.sh and the documentation here are all written for the dense architecture. For a mixture-of-experts model use escha-runtime-qwen3moe, whose defaults are measured on it. A model of a genuinely different architecture will not load — use the matching escha-runtime-<arch> repo.

Quickstart

Full detail, including the per-GPU cookbook and troubleshooting: sglang/INSTALL.md.

python3.12 -m venv .venv && source .venv/bin/activate
pip install -U pip wheel
pip install "torch==2.9.*" --index-url https://download.pytorch.org/whl/cu128   # cu12 torch FIRST
pip install ./sglang/escha-*.whl   # pulls the bundled sglang fork + its full dep closure

hf download EschaLabs/Qwen3.8-27B-Escha-W2 --local-dir ./Qwen3.8-27B-Escha-W2
MODEL=./Qwen3.8-27B-Escha-W2 bash sglang/serve.sh

Then check the stack and the endpoint:

python -c "import torch, escha, sglang; print(torch.cuda.is_available(), hasattr(torch.ops.escha, 'escham_decode_gemv'), escha.__version__)"
curl -s http://127.0.0.1:30000/v1/models | python3 -m json.tool

pip install "torch==2.9.*" is a hard pin, not a suggestion. A bare torch>=2.9 resolves to a newer minor and import escha then fails with undefined symbol: _ZN3c10... — the compiled extension is ABI-linked to libtorch, and that ABI is not stable across PyTorch minors.

Thinking, and why you probably want a budget

This is a reasoning model. With thinking on, the reasoning arrives in reasoning_content and the answer in content — read both, or you will see half the response.

Two per-request levers, both inside chat_template_kwargs (a top-level enable_thinking field is silently ignored):

{ "chat_template_kwargs": {"enable_thinking": true, "reasoning_effort": "xhigh"} }

reasoning_effort is "xhigh" (the default), "medium" or "low"; anything else makes the template raise, which surfaces as an HTTP 400 rather than a silent fallback. It works by injecting one sentence of system instruction — xhigh asks the model to validate assumptions and weigh alternatives, low asks it to keep thinking brief, and medium injects nothing at all, so medium is the neutral, unsteered model rather than a midpoint. It therefore asks for shorter reasoning; it does not bound it. If you are running a benchmark or an agent, set a thinking budget instead, which forces </think> after N reasoning tokens so an answer is always produced: see sglang/INSTALL.md → Bounded thinking and sglang/thinking_budget.py. Without one, the usual failure is finish_reason: "length" with content: null, which a harness scores as wrong rather than as slow.

Requirements

• NVIDIA GPU, compute capability 8.0–12.0 (Ampere → Blackwell), Linux x86-64 with glibc ≥ 2.28. The kernel launch route auto-selects per GPU; you never set it. Per-architecture and per-VRAM launch recipes: sglang/INSTALL.md → Running on your GPU.
• Python 3.12 (the wheel is cp312-only) + CUDA-12 PyTorch 2.9.x. The wheel handles every other dependency.
• A working host C compiler and Python dev headers. Triton JIT-compiles a small shim at CUDA-graph capture time — this is separate from ptxas and from a CUDA toolkit, so "driver only" does not cover it. On slim container images a stripped libisl breaks cc1 while gcc --version still succeeds, and the failure surfaces ~40 s in as a gcc CalledProcessError inside cuda_graph_runner.py — which reads like a runtime bug and is not. Preflight in sglang/INSTALL.md.
• 24 GB VRAM for the shipped defaults (65,536-token context, ~8–9 concurrent streams at short prompts) with a ~10.15 GB model. Those two figures are not simultaneous: the default pool is 68,686 tokens, which is one full-length 64k request or ~8 requests of ~8k. Note also the stream ceiling: MAMBA_RATIO=0.3 sizes the recurrent-state pool, which clamps max_running_requests to 8–9 on a 24 GB card, so the 12/16 entries in the default CUDA_GRAPH_BS are dropped and never captured. To serve more streams raise MAXREQ/MAXMAMBA with MEM — the throughput recipe is in the model card; for long context (128k measured on a 24 GB card) see By VRAM. 16 GB is now measured too, on an RTX 5060 Ti — ~30 tok/s single-stream, and far more context than we had guessed: 110,592 tokens, with an fp8 KV cache doing most of that work. See The 16 GB tier.

Changelog

1.2.0 (2026-08-21) — tensor parallelism (--tp-size N) now works. The escha parameter class pins its own weight loader, which meant sglang's TP slicing never ran and every rank kept the whole checkpoint (rank 0 died with weight must have shape (dim, width)). It now slices per rank, including the fused-on-disk GDN in_proj_qkv, which is split into its three sub-projections first.

Single-GPU users are unaffected. Every new code path is gated on world_size > 1; at --tp-size 1 the loader is byte-for-byte what 1.1.1 did. Verified as an identical shard layout and byte-identical greedy output.

TP > 1 is new and lightly tested — treat it as experimental. It was contributed and validated by @ginerJuanUdesa on 2× RTX 3090 (symmetric 6.02 GB/rank, coherent greedy output). We have one GPU and could not reproduce it, and no numerical equivalence check against --tp-size 1 has been run yet. If you use it for evaluation, sanity-check a benchmark against the single-GPU numbers first. Note that a multi-rank all-reduce reorders float accumulation, so TP > 1 output is not expected to match TP = 1 bit-for-bit even when correct.

On Ampere/Ada/Hopper you can add DETERMINISTIC=1 to remove that reduction-order variance if you want a stricter comparison.

1.1.1 (2026-08-21) — process_weights_after_loading now takes the rank's device instead of a hardcoded cuda:0. The hardcode put every 2-bit buffer on cuda:0 while the input tensor sat on the server's actual device, so any run not on device 0 — --tp-size > 1, or a single-GPU launch with --base-gpu-id N and no CUDA_VISIBLE_DEVICES — hit an illegal memory access on the first forward, behind a traceback that pointed at the kernel rather than at the cause. Bit-identical wherever cuda:0 was already correct, which is every configuration serve.sh ships. Reported with a diagnosis and a fix by @ginerJuanUdesa.

1.1.0 (2026-08-20) — first wheel with the dense (escha) serving path; the 1.0.x wheels registered eschamoe only, so a dense checkpoint failed at registry lookup.

Known limitations

• Serving throughput has been measured on four cards — RTX 5090 (32 GB, sm_120), RTX 4090 (24 GB, sm_89), RTX 3090 (24 GB, sm_86) and RTX 5060 Ti (16 GB, sm_120); all but the 4090 by independent evaluators working only from these docs. The 4090 is also where the model's thinking-on benchmarks (GPQA-Diamond, LiveCodeBench) were produced; its thinking-off commonsense suite ran on an L40. The 40 GB+ tier in the cookbook is still configuration guidance derived from the model size and the wheel's architecture coverage, not a measurement. Per-GPU numbers live on the model card.

• On Ampere the auto-selected kernel route is the slower one at batch 1. ESCHA_ROUTE resolves to lovelace on sm_80/sm_86, but forcing ESCHA_ROUTE=blackwell measured 1.72× faster single-stream on an RTX 3090 (23.6 → 40.7 tok/s, TPOT 42.4 → 24.6 ms) with identical output. The two routes are bit-identical launch geometries, so this is safe to set; the gain is batch-1-only (parity at 2–16). Serving one user on Ampere? Set it.

• DETERMINISTIC=1 fails on consumer Blackwell (sm_120). The deterministic attention kernel requests 104 KB of shared memory per block, above the sm_120 limit, and the server exits during startup. It works on Ampere, Ada and Hopper.

• Greedy output is not bit-reproducible across requests. Batch composition changes fp16 accumulation order, so a near-tie can flip and a long reasoning chain diverges from there. Two identical greedy requests may return different text. Use DETERMINISTIC=1 when you need reproducibility, and never A/B two configurations by diffing one generation.

• CUDA-graph batch sizes are capped at 32, the decode kernel's maximum M (torch.ops.escha.escham_decode_gemv_max_m()). The shipped default list stops at 16 because that is where aggregate throughput peaks on a 4090; capture at 24/32 works and is worth it if you serve that many streams. Past 32 a batch falls through to a large-M path meant for prefill, so the runtime refuses to capture it rather than bake in the wrong kernel.

• ATTN_BACKEND=triton is required on consumer Blackwell (RTX 50-series). The default flashinfer backend asserts on this hybrid architecture at sm_120. The assertion names three acceptable backends — triton, trtllm_mha, fa4 — of which only triton has been run on this model. Note that sm_120 shows steeper long-prompt decode decay than sm_89 (88.5% vs 96.3% of short-prompt rate at a 5,000-token prompt); the attention path is the obvious suspect and nobody has run the A/B that would confirm it.

• The model's stock chat template rejects the developer role. It accepts system, user, assistant and tool; anything else raises Unexpected message role. and surfaces as an HTTP 400 from the template, not as a server fault. Several OpenAI-compatible agent harnesses send role: "developer" for their system prompt and so fail every request — map it to system client-side, or patch the template. The same template also raises on genuinely malformed conversations (a missing user query, a system message in the wrong position) and on a reasoning_effort outside xhigh/medium/low. All of this is the base model's own template, carried over unmodified.

• One report of a GPU falling off the bus under sustained load, on a passed-through card. @JacksonAtkinsX hit NVRM: Xid ... 79, GPU has fallen off the bus four times out of four on an RTX 5060 Ti 16 GB, 12–18 minutes into a sustained single-stream agent workload, and each time the card needed a full host power cycle — FLR, secondary-bus reset and remove/rescan all reported success while the device stayed in D3cold. It reproduced across two driver branches, two runtime versions, both KV dtypes and three context lengths, in prefill and in decode alike. The card is passed through to a VM with vfio.

  We have not reproduced anything like it on the 3090, 4090, 5090 or L40 across the campaigns behind these docs, none of them on a passed-through GPU. Xid 79 is a bus-level fault: a misbehaving CUDA kernel produces an Xid 13/31/43 or an MMU fault and leaves the device resettable, whereas here no reset short of a power cycle worked, which is a platform signature rather than a userspace one. Two hypotheses that report could not close, and what we would check:

  • Run any other sustained 100% load for 30 minutes on the same box — gpu-burn, or a different model on a stock runtime. This is the one test that settles it: if that also falls off the bus, the runtime is not involved.
  • Power transients. nvidia-smi was sampled at 5 s, which cannot see the sub-millisecond spikes that trip a PSU's over-current protection, so "166 W under a 180 W cap with no throttle flags" does not rule power out. Log power.draw.instant and temperature.memory at 100 ms, and check PSU headroom on the rail feeding the card.
  • PCIe power management under vfio, which the D3cold-and-never-returns behaviour fits: vfio-pci disable_idle_d3=1, and pcie_aspm=off pcie_port_pm=off on the host command line.

  One thing we will say on our own side: this runtime's 2-bit decode path is arithmetic-heavy and holds a high, flat ALU duty cycle rather than the burstier profile of an fp16 matmul, so it is a harsher electrical stressor than most inference workloads. That is a reason to want power headroom before a long unattended run, not evidence of a defect.

License

Everything here is released under the Apache License, Version 2.0 — see LICENSE. All bundled third-party code is permissive (Apache-2.0 / MIT / BSD-3-Clause) — no copyleft. Full texts and the component inventory: THIRD_PARTY_LICENSES/. Model weights are not in this repo and carry their own license in the model repository.