metadata
title: 88plug
88plug
Post-training quantization for production GPU inference
What we publish
Canonical catalog: 48 repos · product_done: 10/48 (gold/provisional + smoke=pass + HF).
Compressed-tensors models quantized for native vLLM deployment. No extra flags, no monkey-patching — weights load automatically on any Ampere+ GPU.
| Format | Precision | Method | Use case |
|---|---|---|---|
W8A16 |
INT8 weights · BF16 activations | compressed-tensors PTQ | Max throughput, near-lossless quality |
W4A16 |
INT4 weights · BF16 activations | AutoRound (iters=200) | Lowest VRAM footprint, high accuracy |
Quick start
pip install vllm>=0.21.0
vllm serve 88plug/<model-name> --dtype bfloat16
No --quantization flag required. The compressed-tensors config is embedded in the model weights
and auto-detected by vLLM v0.21.0+.
Hardware
- Quantized on NVIDIA RTX A6000 (48 GB)
- Tested on Ampere, Ada Lovelace, and Hopper architectures
- W8A16 models require ~50% the VRAM of BF16
- W4A16 models require ~25% the VRAM of BF16
Stack
- vLLM — inference engine
- llm-compressor — quantization pipeline
- AutoRound — W4A16 calibration (iters=200)
- compressed-tensors — weight format