Instructions to use amd/DeepSeek-V4-Pro-MXFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use amd/DeepSeek-V4-Pro-MXFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="amd/DeepSeek-V4-Pro-MXFP4")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("amd/DeepSeek-V4-Pro-MXFP4")
model = AutoModelForCausalLM.from_pretrained("amd/DeepSeek-V4-Pro-MXFP4", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use amd/DeepSeek-V4-Pro-MXFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "amd/DeepSeek-V4-Pro-MXFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/DeepSeek-V4-Pro-MXFP4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/amd/DeepSeek-V4-Pro-MXFP4

SGLang

How to use amd/DeepSeek-V4-Pro-MXFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "amd/DeepSeek-V4-Pro-MXFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/DeepSeek-V4-Pro-MXFP4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "amd/DeepSeek-V4-Pro-MXFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/DeepSeek-V4-Pro-MXFP4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use amd/DeepSeek-V4-Pro-MXFP4 with Docker Model Runner:
```
docker model run hf.co/amd/DeepSeek-V4-Pro-MXFP4
```

DeepSeek-V4-Pro-MXFP4

Model Overview

Model Architecture: DeepseekV4ForCausalLM
- Input: Text
- Output: Text
Supported Hardware Microarchitecture: AMD MI355 / MI350 (gfx950)
ROCm: 7.2.0
PyTorch: 2.9.1
Transformers: 5.13.1
Operating System(s): Linux
Inference Engine: vLLM
Model Optimizer: AMD-Quark (v0.12.0)
- Quantized layers: All routed + shared MoE expert projections. All other modules (attention, the MoE router gate, norms, embeddings, the output head, and the MTP block) are excluded and kept in original precision.
- Weight quantization: OCP MXFP4, Static
- Activation quantization: OCP MXFP4, Dynamic

Model Quantization

Quantized from deepseek-ai/DeepSeek-V4-Pro with AMD Quark. The pipeline re-quantizes only the MoE expert weights and activations to MXFP4. All non-expert modules are kept as-is via the exclude list.

Quantization script

from quark.torch import ModelQuantizer
from quark.torch.quantization.config.template import LLMTemplate

template = LLMTemplate.get('deepseek_v4')
qconfig = template.get_config(scheme='mxfp4')
ModelQuantizer(qconfig).direct_quantize_checkpoint(
    pretrained_model_path='<DSV4_Pro_src_path>',
    save_path='<output_dir>',
    keep_excluded_layers_as_original_model_state=True,
)

Deployment

Use with vLLM

This model can be deployed efficiently using the vLLM backend based on the Docker image rocm/vllm-dev:nightly_main_20260714. vLLM and lm_eval are both installed from source.

Evaluation

The model was evaluated on gsm8k (8-shot) benchmark using the vLLM framework.

Accuracy

Benchmark	deepseek-ai/DeepSeek-V4-Pro	amd/DeepSeek-V4-Pro-MXFP4	Recovery
GSM8K (strict-match)	94.90	93.6	99.0%

Reproduction

The GSM8K results were obtained using the lm-eval framework, based on the Docker image rocm/vllm-dev:nightly_main_20260714.

Launching server

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
vllm serve amd/DeepSeek-V4-Pro-MXFP4 --tensor-parallel-size 4 --kv-cache-dtype fp8 \
  --trust-remote-code --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 --enable-auto-tool-choice \
  --compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}'

Evaluating model in a new terminal

lm_eval --model local-completions \
    --model_args model=amd/DeepSeek-V4-Pro-MXFP4,base_url=http://localhost:30000/v1/completions,tokenized_requests=False,num_concurrent=32 \
    --tasks gsm8k --batch_size auto --num_fewshot 8

License

This model is a quantized derivative of deepseek-ai/DeepSeek-V4-Pro and is distributed under the same license as the source model: the MIT License. A copy of the upstream LICENSE is included in this repository.

Modifications Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. AMD has modified the model weights of the MoE expert layers by quantizing them to MXFP4 with AMD Quark; the modifications are provided under the same MIT License and are not subject to any separate or different license.

Downloads last month: 25

Safetensors

Model size

860B params

Tensor type

F32

BF16

F8_E8M0

F8_E4M3

I64

Model tree for amd/DeepSeek-V4-Pro-MXFP4

Base model

deepseek-ai/DeepSeek-V4-Pro

Quantized

(25)

this model