Instructions to use tencent/Hy3-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tencent/Hy3-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tencent/Hy3-FP8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tencent/Hy3-FP8")
model = AutoModelForCausalLM.from_pretrained("tencent/Hy3-FP8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tencent/Hy3-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tencent/Hy3-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tencent/Hy3-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tencent/Hy3-FP8

SGLang

How to use tencent/Hy3-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tencent/Hy3-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tencent/Hy3-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tencent/Hy3-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tencent/Hy3-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use tencent/Hy3-FP8 with Docker Model Runner:
```
docker model run hf.co/tencent/Hy3-FP8
```

vllm 0.24 RuntimeError: HYV3ReasoningParser reasoning parser could not locate think start/end tokens in the tokenizer!

by AImhotep - opened 3 days ago

Discussion

AImhotep

3 days ago

(APIServer pid=143) Traceback (most recent call last):
(APIServer pid=143) File "/root/vllm/.venv/bin/vllm", line 10, in
(APIServer pid=143) sys.exit(main())
(APIServer pid=143) ^^^^^^
(APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 95, in main
(APIServer pid=143) args.dispatch_function(args)
(APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 148, in cmd
(APIServer pid=143) uvloop.run(run_server(args))
(APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/uvloop/init.py", line 96, in run
(APIServer pid=143) return __asyncio.run(
(APIServer pid=143) ^^^^^^^^^^^^^^
(APIServer pid=143) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=143) return runner.run(main)
(APIServer pid=143) ^^^^^^^^^^^^^^^^
(APIServer pid=143) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=143) return self._loop.run_until_complete(task)
(APIServer pid=143) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=143) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/uvloop/init.py", line 48, in wrapper
(APIServer pid=143) return await main
(APIServer pid=143) ^^^^^^^^^^
(APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 663, in run_server
(APIServer pid=143) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 677, in run_server_worker
(APIServer pid=143) async with build_async_engine_client(
(APIServer pid=143) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=143) return await anext(self.gen)
(APIServer pid=143) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 99, in build_async_engine_client
(APIServer pid=143) async with build_async_engine_client_from_engine_args(
(APIServer pid=143) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=143) return await anext(self.gen)
(APIServer pid=143) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 123, in build_async_engine_client_from_engine_args
(APIServer pid=143) vllm_config = engine_args.create_engine_config(usage_context=usage_context)
(APIServer pid=143) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/vllm/engine/arg_utils.py", line 2316, in create_engine_config
(APIServer pid=143) config = VllmConfig(
(APIServer pid=143) ^^^^^^^^^^^
(APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/pydantic/_internal/_dataclasses.py", line 121, in init
(APIServer pid=143) s.pydantic_validator.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
(APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/vllm/config/vllm.py", line 1450, in post_init
(APIServer pid=143) self.reasoning_config.initialize_token_ids(self.model_config)
(APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/vllm/config/reasoning.py", line 80, in initialize_token_ids
(APIServer pid=143) reasoning_parser = parser_cls(tokenizer)
(APIServer pid=143) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/vllm/reasoning/hy_v3_reasoning_parser.py", line 29, in init
(APIServer pid=143) super().init(tokenizer, *args, **kwargs)
(APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/vllm/reasoning/basic_parsers.py", line 65, in init
(APIServer pid=143) raise RuntimeError(
(APIServer pid=143) RuntimeError: HYV3ReasoningParser reasoning parser could not locate think start/end tokens in the tokenizer!

stevenkuang

Tencent org 2 days ago

Could you try the latest code or the vllm-openai:hy3 image?

AImhotep

2 days ago

On vllm-openai:hy3 it works perfectly fine, thank you.

I wonder why it isn't on the nightly version vllm that I've compiled - but this is something to be resolved later :)

AImhotep changed discussion status to closed 2 days ago

stevenkuang

Tencent org 2 days ago

You can check whether this fix is included: https://github.com/vllm-project/vllm/pull/47192. If so, vLLM will work properly.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment