88plug

Post-training quantization for production GPU inference

What we publish

Canonical catalog: 48 repos · product_done: 10/48 (gold/provisional + smoke=pass + HF).

Compressed-tensors models quantized for native vLLM deployment. No extra flags, no monkey-patching — weights load automatically on any Ampere+ GPU.

Format	Precision	Method	Use case
`W8A16`	INT8 weights · BF16 activations	compressed-tensors PTQ	Max throughput, near-lossless quality
`W4A16`	INT4 weights · BF16 activations	AutoRound (iters=200)	Lowest VRAM footprint, high accuracy

Quick start

pip install vllm>=0.21.0

vllm serve 88plug/<model-name> --dtype bfloat16

No --quantization flag required. The compressed-tensors config is embedded in the model weights and auto-detected by vLLM v0.21.0+.

Hardware

Quantized on NVIDIA RTX A6000 (48 GB)
Tested on Ampere, Ada Lovelace, and Hopper architectures
W8A16 models require ~50% the VRAM of BF16
W4A16 models require ~25% the VRAM of BF16

Stack

vLLM — inference engine
llm-compressor — quantization pipeline
AutoRound — W4A16 calibration (iters=200)
compressed-tensors — weight format