chatterbox-turbo-lora

LoRA-aware ONNX export of the Chatterbox-Turbo language-model subgraph, quantized to q4f16 for on-device inference (Android / iOS via ORT).

This is the base LM only — the full Chatterbox-Turbo TTS pipeline still needs the conditional-encoder, speech-token, and S3-Gen subgraphs from the upstream Chatterbox repo.

What's new in v2.0.0

v1.0.0 of the language-model subgraph had no LoRA input ports — adapter weights could not be bound at runtime. v2.0.0 wraps every Conv1D projection inside the transformer (attn.c_attn, attn.c_proj, mlp.c_fc, mlp.c_proj) in a LoraConv1D that exposes two extra input tensors per layer:

lora.layers.{N}.{attn_c_attn|attn_c_proj|mlp_c_fc|mlp_c_proj}.{A|B}

→ 24 layers × 4 projections × 2 tensors = 192 extra graph inputs.

At inference time the host runtime binds a PEFT-trained LoRA adapter (rank, scaling baked in) to these inputs via the ONNX Runtime OrtLoraAdapter API. Unbound inputs default to zero-sized initializers so the graph still runs without any adapter loaded — the base voice synthesizes as-is.

Files

File	Size	Purpose
language_model_q4f16.onnx	~0.8 MB	Graph protobuf (no weights)
language_model_q4f16.onnx_data	~222 MB	External weight data (q4f16)

Both files must be downloaded together. The .onnx file references the .onnx_data companion by relative filename.

Quantization details

• Format: q4f16 (4-bit asymmetric weights + fp16 scales)
• Block size: 32 (matches Chatterbox's original 1.0.0 layout)
• Quantizer: onnxruntime.quantization.MatMulNBitsQuantizer
• Excluded from quantization: all 192 LoRA-path MatMul/Add nodes (kept fp32 so adapter weights can be hot-swapped without retraining scales)
• The 96 GPT-2 projection layers are first rewritten from Gemm → MatMul + Add so they fall inside the quantizer's coverage; without this step the file would stay at ~1.2 GB.

Embedding tables (wte, wpe) remain fp32 because they are read via Gather, not MatMul, and MatMulNBitsQuantizer does not touch them.

License

MIT, inherited from the upstream Chatterbox project.