Instructions to use mwitiderrick/open_llama_3b_instruct_v_0.2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mwitiderrick/open_llama_3b_instruct_v_0.2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mwitiderrick/open_llama_3b_instruct_v_0.2")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mwitiderrick/open_llama_3b_instruct_v_0.2")
model = AutoModelForCausalLM.from_pretrained("mwitiderrick/open_llama_3b_instruct_v_0.2")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use mwitiderrick/open_llama_3b_instruct_v_0.2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mwitiderrick/open_llama_3b_instruct_v_0.2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mwitiderrick/open_llama_3b_instruct_v_0.2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/mwitiderrick/open_llama_3b_instruct_v_0.2

SGLang

How to use mwitiderrick/open_llama_3b_instruct_v_0.2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mwitiderrick/open_llama_3b_instruct_v_0.2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mwitiderrick/open_llama_3b_instruct_v_0.2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mwitiderrick/open_llama_3b_instruct_v_0.2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mwitiderrick/open_llama_3b_instruct_v_0.2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use mwitiderrick/open_llama_3b_instruct_v_0.2 with Docker Model Runner:
```
docker model run hf.co/mwitiderrick/open_llama_3b_instruct_v_0.2
```

OpenLLaMA Instruct: An Open Reproduction of LLaMA

This is an OpenLlama model that has been fine-tuned on 1 epoch of the Open-Platypus dataset.

The modified version of the dataset can be found here

Prompt Template

### Instruction:

{query}

### Response:
<Leave new line for model to respond>

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM,pipeline

tokenizer = AutoTokenizer.from_pretrained("mwitiderrick/open_llama_3b_instruct_v_0.2")
model = AutoModelForCausalLM.from_pretrained("mwitiderrick/open_llama_3b_instruct_v_0.2")
query = "Provide step-by-step instructions for making a sweet chicken bugger"
text_gen = pipeline(task="text-generation", model=model, tokenizer=tokenizer, max_length=500)
output = text_gen(f"### Instruction:\n{query}\n### Response:\n")
print(output[0]['generated_text'])
"""
### Instruction:
Provide step-by-step instructions for making a sweet chicken bugger
### Response:
Step 1: Gather your ingredients
1. 1/2 cup of sugar
2. 1/2 cup of corn syrup
3. 1/2 cup of water
4. 1/2 cup of vegetable oil
5. 1/2 cup of vanilla extract
6. 1/2 cup of baking soda
7. 1/2 cup of salt
8. 1/2 cup of flour
9. 1/2 cup of milk
10. 1/2 cup of egg whites

Step 2: Mix the ingredients together
1. Combine the sugar, corn syrup, water, vegetable oil, vanilla extract, baking soda, and salt in a large bowl.
2. Whisk together until smooth.
3. Add the flour and mix until combined.
4. Add the milk and egg whites and mix until combined.
5. Pour the mixture into a greased 9x13 inch baking pan.
6. Bake for 30 minutes or until a toothpick inserted into the center comes out clean.

Step 3: Make the chicken bugger
1. Preheat the oven to 350 degrees Fahrenheit.
2. In a large bowl, combine the corn syrup, sugar, and cornstarch.
3. Add the chicken and mix well.
4. Divide the mixture into 12 equal portions and shape each portion into a chicken shape.
5. Place the chicken shapes on a baking sheet lined with parchment paper.
6. Bake for 15 minutes or until the chicken is cooked through.
7. Remove the chicken from the oven and allow to cool for 5 minutes.
8. Using a fork, carefully remove the chicken from the shells and place on a serving platter.
9. Serve with a side of gravy.

Step 4: Make the gravy
1. In a small saucepan, combine the cornstarch and water.
2. Stir until the mixture is smooth and begins to thicken.
3. Add the chicken broth and bring to a boil.
4. Reduce the heat to low and simmer for 10 minutes or until the gravy is
"""

TruthfulQA metrics



|  Groups  |Version|Filter|n-shot|  Metric   | Value  |   |Stderr|
|----------|-------|------|-----:|-----------|-------:|---|-----:|
|truthfulqa|N/A    |none  |     0|acc        |  0.3166|±  |0.0012|
|          |       |none  |     0|bleu_max   | 23.7766|±  |0.7660|
|          |       |none  |     0|bleu_acc   |  0.3207|±  |0.0163|
|          |       |none  |     0|bleu_diff  | -7.1853|±  |0.7396|
|          |       |none  |     0|rouge1_max | 48.6534|±  |0.8706|
|          |       |none  |     0|rouge1_acc |  0.2766|±  |0.0157|
|          |       |none  |     0|rouge1_diff| -9.8011|±  |0.7883|
|          |       |none  |     0|rouge2_max | 31.9289|±  |0.9637|
|          |       |none  |     0|rouge2_acc |  0.2399|±  |0.0149|
|          |       |none  |     0|rouge2_diff|-11.3958|±  |0.9220|
|          |       |none  |     0|rougeL_max | 45.4592|±  |0.8754|
|          |       |none  |     0|rougeL_acc |  0.2754|±  |0.0156|
|          |       |none  |     0|rougeL_diff|-10.0740|±  |0.7807|

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	38.97
AI2 Reasoning Challenge (25-Shot)	38.48
HellaSwag (10-Shot)	66.77
MMLU (5-Shot)	25.34
TruthfulQA (0-shot)	38.16
Winogrande (5-shot)	63.46
GSM8k (5-shot)	1.59

Downloads last month: 15

Safetensors

Model size

3B params

Tensor type

F16

Model tree for mwitiderrick/open_llama_3b_instruct_v_0.2

Base model

openlm-research/open_llama_3b

Finetuned

(17)

this model

Dataset used to train mwitiderrick/open_llama_3b_instruct_v_0.2

Evaluation results

hellaswag(0-Shot) on hellaswag
self-reported

0.488
winogrande(0-Shot) on winogrande
self-reported

0.613
arc_challenge(0-Shot) on arc_challenge
open_llama_3b_instruct_v_0.2 model card

0.336
normalized accuracy on AI2 Reasoning Challenge (25-Shot)
test set Open LLM Leaderboard

38.480
normalized accuracy on HellaSwag (10-Shot)
validation set Open LLM Leaderboard

66.770
accuracy on MMLU (5-Shot)
test set Open LLM Leaderboard

25.340
mc2 on TruthfulQA (0-shot)
validation set Open LLM Leaderboard

38.160
accuracy on Winogrande (5-shot)
validation set Open LLM Leaderboard

63.460
accuracy on GSM8k (5-shot)
test set Open LLM Leaderboard

1.590