Instructions to use skumar9/Llama-medx_v3.2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use skumar9/Llama-medx_v3.2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="skumar9/Llama-medx_v3.2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("skumar9/Llama-medx_v3.2")
model = AutoModelForCausalLM.from_pretrained("skumar9/Llama-medx_v3.2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use skumar9/Llama-medx_v3.2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "skumar9/Llama-medx_v3.2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "skumar9/Llama-medx_v3.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/skumar9/Llama-medx_v3.2

SGLang

How to use skumar9/Llama-medx_v3.2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "skumar9/Llama-medx_v3.2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "skumar9/Llama-medx_v3.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "skumar9/Llama-medx_v3.2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "skumar9/Llama-medx_v3.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use skumar9/Llama-medx_v3.2 with Docker Model Runner:
```
docker model run hf.co/skumar9/Llama-medx_v3.2
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

Built for the Medical Domain:

Llama-medx_v3.2 is specifically fine-tuned for medical domain.

Training Process:

Built on the Meta-Llama-3.1-8B architecture. Llama-medx_v3.2 has been finetuned using Supervised Fine-Tuning (SFT) and Odds Ratio Preference Optimization (ORPO). This ensures that the model aligns effectively with medical terminology and reasoning while maintaining learning efficiency. Hyperparameter tuning strategies have been carefully implemented to prevent catastrophic forgetting, ensuring consistent performance across various tasks. Benchmarked model was further finetuned for anwering mcq questions.

Data Preparation:

Publically available datasets, Enriched datasets for reinforcement feedback where the llama base model's knowledge and accuracy was not upto the mark.

Use with transformers

Please ensure transformers>=4.45.2

import torch
import transformers

model_id = "skumar9/Llama-medx_v3.2"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

prompt = "Question: A 20-year-old man comes to the physician because of worsening gait unsteadiness and bilateral hearing loss for 1 month. He has had intermittent tingling sensations on both cheeks over this time period. He has no history of serious medical illness and takes no medications. Audiometry shows bilateral sensorineural hearing loss. Genetic evaluation shows a mutation of a tumor suppressor gene on chromosome 22 that encodes merlin. This patient is at increased risk for which of the following conditions?\nA. Renal cell carcinoma\nB. Meningioma\nC. Astrocytoma\nD. Vascular malformations\nAnswer:\n"
gen_kwargs = {
    "return_full_text": False,
    "max_new_tokens": 100,
}
print(pipeline(prompt, **gen_kwargs))

Downloads last month: 10

Safetensors

Model size

8B params

Tensor type

F16

Model tree for skumar9/Llama-medx_v3.2

Merges

2 models

Quantizations

6 models