shreyan35/Qwopus3.6-27B-Coder-mlx-6Bit

Qwopus 3.6 27B Coder (6-bit MLX Quantization)

This repository hosts a high-performance 6-bit MLX quantization of Jackrong/Qwopus3.6-27B-Coder, converted using mlx-lm (v0.31.2).

Optimized specifically for Apple Silicon (M-series chips), this model balances the exceptional coding intelligence of the 27B parameter Qwopus architecture with the memory and speed efficiency required for local, low-latency deployment.

Key Features

• TurboQuant Accelerated: Leverages advanced MLX kernel optimizations for accelerated matrix multiplication, ensuring faster token generation rates during local execution, making the kv cache at the models whopping full 262k context just around ~3-4gb from ~180gb
• ~50% Memory Reduction: Cuts VRAM usage nearly in half compared to the base 16-bit model, opening up local execution on mid-tier unified memory configurations without spilling into system swap.
• Near-Lossless Precision: Grouped 6-bit quantization maintains the original model's structural code generation capabilities, logic, and syntax proficiency with negligible degradation.
• Apple Silicon Native: Designed from the ground up for unified memory architectures, leveraging direct hardware acceleration via the MLX framework.

---

Installation

Ensure you have the latest version of the MLX language model library installed:

pip install mlx-lm

Quick Start

You can load and run inference with this model locally using the following Python script. It automatically detects and applies the correct chat template for structured instruction-following.

from mlx_lm import load, generate

# Load the optimized 6-bit model and its tokenizer
model, tokenizer = load("shreyan35/Qwopus3.6-27B-Coder-mlx-6Bit")

# Define your programming task or prompt
prompt = "Write an optimized Python function to find the longest palindromic substring."

# Apply the model's native chat template if available
if hasattr(tokenizer, "apply_chat_template") and tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )

# Generate the response with real-time streaming/verbosity
response = generate(model, tokenizer, prompt=prompt, verbose=True)

Acknowledgements

• Base Model: Credit to Jackrong for the original Qwopus3.6-27B-Coder architecture.
• Infrastructure: The Apple Machine Learning Research team for the ongoing development of the MLX framework.