poolside

/

Laguna-S-2.1-INT4

like 25

Follow

Poolside 1.33k

Hi,
With which version of vLLM and which parameters (temperature, top_p...) and on which platform (Dgx Spark, ...) did you manage to make it work?
You must be the only one I feel because all the others including me can not get anything out of it but there may be something subtle that poses problems.
Have you tried to ask him something very complex?
Thank you 🙂

Hey so i have 4x amd R9700 ai pro gpus I was able to set the context to 1 million so thats why I used auto and this image i use is special for my gpus it fixes an issue with slower context speed as it builds here is my config top is aml731/vllm-aiter:v0.24.0 and bottom is the newest vllm/vllm-openai-rocm:latest a few things changed and maybe somethings i dont need im a noob and had ai help me but this is hours and hours of small test. also if you want to go higher than the 256k i used

--override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":20}'

and it worked. just now i want to find out the different in intelligence from FP8 vs INT4 and was unable to get the poolside/Laguna-S-2.1-DFlash-INT4 think it might be an amd issue

also with vllm/vllm-openai-rocm:latest i had to put

  NCCL_P2P_DISABLE: "1"
  PYTORCH_ALLOC_CONF: "expandable_segments:True"
  NCCL_PROTO: "Simple"

not sure what it does the speeds are pretty much the same but without it this would crash for me atleast

==========================================================================================

services:
vllm-laguna:
image: aml731/vllm-aiter:v0.24.0
container_name: vllm-laguna
restart: unless-stopped
network_mode: host
ipc: host
group_add:
- "44"
- "993"
cap_add:
- SYS_PTRACE
security_opt:
- seccomp=unconfined
devices:
- /dev/kfd:/dev/kfd
- /dev/dri:/dev/dri
volumes:
- /home/models/gpu/poolside:/models
- /home/models/gpu/poolside/vllm-cache:/root/.cache
environment:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_ALLOW_RDNA4_AITER_ATTENTION: "1"
VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION: "1"
VLLM_ROCM_USE_AITER_MHA: "1"
PYTORCH_ALLOC_CONF: "expandable_segments:True"
HIP_VISIBLE_DEVICES: "0,1,2,3"
command: >
python3 -m vllm.entrypoints.openai.api_server
--host 0.0.0.0 --port 8000
--model /models/Laguna-S-2.1-INT4
--served-model-name laguna-s-2.1
--trust-remote-code
--tensor-parallel-size 4
--dtype auto
--attention-backend ROCM_AITER_UNIFIED_ATTN
--compilation-config '{"pass_config":{"fuse_norm_quant":false}}'
--default-chat-template-kwargs '{"enable_thinking": true}'
--gpu-memory-utilization 0.98
--max-num-batched-tokens 8192
--max-model-len auto
--max-num-seqs 1
--enable-prefix-caching
--reasoning-parser poolside_v1
--tool-call-parser poolside_v1
--enable-auto-tool-choice
--safetensors-load-strategy=prefetch
--override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":20}'

==========================================================================================

logs

(Worker_TP0 pid=264) INFO 07-24 22:09:00 [monitor.py:81] Initial profiling/warmup run took 2.08 s

(Worker_TP0 pid=264) INFO 07-24 22:09:06 [gpu_worker.py:508] Available KV cache memory: 11.61 GiB

(EngineCore pid=186) INFO 07-24 22:09:06 [kv_cache_utils.py:1946] Auto-fit max_model_len: full model context length 1048576 fits in available GPU memory

(EngineCore pid=186) INFO 07-24 22:09:06 [kv_cache_utils.py:2146] GPU KV cache size: 1,979,464 tokens

(EngineCore pid=186) INFO 07-24 22:09:06 [kv_cache_utils.py:2147] Maximum concurrency for 1,048,576 tokens per request: 1.89x

[ 1,000] warmup 1/3 ... ctx= 1,034 ttft=0.2s prefill=4419 tok/s decode=75.0 tok/s 737W
[ 1,000] measure 2/3 ... ctx= 1,033 ttft=0.2s prefill=5724 tok/s decode=75.3 tok/s 804W
[ 1,000] measure 3/3 ... ctx= 1,033 ttft=0.2s prefill=5680 tok/s decode=74.9 tok/s 811W
[ 8,000] warmup 1/3 ... ctx= 8,032 ttft=1.1s prefill=7274 tok/s decode=74.6 tok/s 785W
[ 8,000] measure 2/3 ... ctx= 8,031 ttft=1.1s prefill=7319 tok/s decode=74.6 tok/s 806W
[ 8,000] measure 3/3 ... ctx= 8,032 ttft=1.1s prefill=7293 tok/s decode=74.4 tok/s 790W
[ 32,000] warmup 1/3 ... ctx= 32,026 ttft=4.9s prefill=6507 tok/s decode=73.2 tok/s 818W
[ 32,000] measure 2/3 ... ctx= 32,026 ttft=4.9s prefill=6531 tok/s decode=73.1 tok/s 795W
[ 32,000] measure 3/3 ... ctx= 32,026 ttft=4.9s prefill=6536 tok/s decode=73.1 tok/s 872W
[ 64,000] warmup 1/3 ... ctx= 64,018 ttft=11.3s prefill=5668 tok/s decode=71.8 tok/s 816W
[ 64,000] measure 2/3 ... ctx= 64,017 ttft=11.3s prefill=5669 tok/s decode=71.8 tok/s 850W
[ 64,000] measure 3/3 ... ctx= 64,017 ttft=11.3s prefill=5674 tok/s decode=71.7 tok/s 818W
[128,000] warmup 1/3 ... ctx=128,002 ttft=28.7s prefill=4453 tok/s decode=70.0 tok/s 854W
[128,000] measure 2/3 ... ctx=128,002 ttft=28.7s prefill=4460 tok/s decode=69.9 tok/s 857W

==========================================================================================

services:
vllm-laguna:
image: vllm/vllm-openai-rocm:latest
container_name: vllm-laguna
restart: "no"
network_mode: host
ipc: host
group_add:
- "44"
- "993"
cap_add:
- SYS_PTRACE
security_opt:
- seccomp=unconfined
devices:
- /dev/kfd:/dev/kfd
- /dev/dri:/dev/dri
volumes:
- /home/models/gpu/poolside:/models
- /home/models/gpu/poolside/vllm-cache:/root/.cache
environment:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_ALLOW_RDNA4_AITER_ATTENTION: "1"
VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION: "1"
VLLM_ROCM_USE_AITER_MOE: "1"
VLLM_ROCM_USE_AITER_RMSNORM: "0"
VLLM_ROCM_USE_AITER_MHA: "1"
NCCL_P2P_DISABLE: "1"
PYTORCH_ALLOC_CONF: "expandable_segments:True"
NCCL_PROTO: "Simple"
HIP_VISIBLE_DEVICES: "0,1,2,3"
entrypoint: ["vllm", "serve"]
command: >
/models/Laguna-S-2.1-INT4
--host 0.0.0.0 --port 8000
--served-model-name laguna-s-2.1
--tensor-parallel-size 4
--dtype auto
--compilation-config '{"pass_config":{"fuse_norm_quant":false}}'
--default-chat-template-kwargs '{"enable_thinking": true}'
--attention-backend ROCM_AITER_UNIFIED_ATTN
--gpu-memory-utilization 0.95
--max-num-batched-tokens 8192
--max-model-len auto
--max-num-seqs 1
--enable-prefix-caching
--trust-remote-code
--reasoning-parser poolside_v1
--tool-call-parser poolside_v1
--enable-auto-tool-choice
--safetensors-load-strategy=prefetch
--override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":20}'

==========================================================================================

logs

(Worker_TP0 pid=322) INFO 07-24 22:19:33 [monitor.py:81] Initial profiling/warmup run took 7.83 s

(EngineCore pid=218) INFO 07-24 22:19:33 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).

(Worker_TP0 pid=322) INFO 07-24 22:19:36 [gpu_worker.py:538] Available KV cache memory: 10.44 GiB

(EngineCore pid=218) INFO 07-24 22:19:36 [kv_cache_utils.py:1946] Auto-fit max_model_len: full model context length 1048576 fits in available GPU memory

(EngineCore pid=218) INFO 07-24 22:19:36 [kv_cache_utils.py:2146] GPU KV cache size: 1,778,177 tokens

(EngineCore pid=218) INFO 07-24 22:19:36 [kv_cache_utils.py:2147] Maximum concurrency for 1,048,576 tokens per request: 1.70x

==========================================================================================

[ 1,000] warmup 1/3 ... ctx= 1,033 ttft=2.3s prefill=455 tok/s decode=70.5 tok/s 760W
[ 1,000] measure 2/3 ... ctx= 1,033 ttft=0.2s prefill=4291 tok/s decode=71.3 tok/s 818W
[ 1,000] measure 3/3 ... ctx= 1,033 ttft=0.2s prefill=4242 tok/s decode=71.7 tok/s 836W
[ 8,000] warmup 1/3 ... ctx= 8,032 ttft=1.6s prefill=5140 tok/s decode=70.6 tok/s 769W
[ 8,000] measure 2/3 ... ctx= 8,032 ttft=1.6s prefill=5155 tok/s decode=70.5 tok/s 777W
[ 8,000] measure 3/3 ... ctx= 8,032 ttft=1.6s prefill=5148 tok/s decode=70.6 tok/s 785W
[ 32,000] warmup 1/3 ... ctx= 32,026 ttft=6.7s prefill=4781 tok/s decode=68.6 tok/s 767W
[ 32,000] measure 2/3 ... ctx= 32,026 ttft=6.7s prefill=4805 tok/s decode=68.3 tok/s 778W
[ 32,000] measure 3/3 ... ctx= 32,026 ttft=6.7s prefill=4805 tok/s decode=68.5 tok/s 830W
[ 64,000] warmup 1/3 ... ctx= 64,018 ttft=14.9s prefill=4305 tok/s decode=66.6 tok/s 851W
[ 64,000] measure 2/3 ... ctx= 64,016 ttft=14.9s prefill=4309 tok/s decode=66.2 tok/s 867W
[ 64,000] measure 3/3 ... ctx= 64,018 ttft=14.8s prefill=4316 tok/s decode=66.7 tok/s 804W
[128,000] warmup 1/3 ... ctx=128,002 ttft=35.8s prefill=3575 tok/s decode=63.3 tok/s 883W
[128,000] measure 2/3 ... ctx=128,002 ttft=35.8s prefill=3577 tok/s decode=63.6 tok/s 835W

@zxrockbearxz
have you tried spec decode? --speculative-config '{"model":"poolside/Laguna-S-2.1-DFlash-INT4","num_speculative_tokens":15,"method":"dflash"}'

ive tried when using DFlash i get 20 tokens and from what i can find online is that its broken on amd right now

I have 4 R9700 and was about to try to set up INT4 via vLLM. @zxrockbearxz should I wait for the software to mature before I jump in? I've been seeing people have a lot of issues with looping. And the Q4 in llama.cpp overthinks everything.

Yes, it is better to wait, it is not at all at the point and everything suggests that the root concern comes from the base model... But if you have time to help them find the worry you can 🙂 Then you can try it by disabling the reasoning but it will not be at all at the level they announced on the benchmarks.

@schimazing I mean i use this to build a small project for work I have had it looping once. my gpus were 200/300 watts reset the portainer and they stopped was really weird. I was using qwen 3.7 27b I have had some good experiences with this model or at least its been pretty good. in hermes agent thats where i use it. hopefully with time they work out the bugs would like to see more independent reviewers but glad to see something different in the space and american model that we can download.