Luna-0.1b-Instruct

This is a 124 Million parameter language model trained from scratch. Structurally identical to the original OpenAI GPT-2 Small, this model represents a complete end-to-end LLM training pipeline built independently.

🧠 Training Details

The model was trained in two distinct phases to achieve "Compute-Optimal" performance for its size:

1. Base Pretraining

• Dataset: Fineweb-Edu (High-quality educational text).
• Tokens: ~2.6 Billion tokens.

2. Supervised Fine-Tuning (SFT)

• Dataset: UltraChat_200k (Instruction / Q&A pairs).
• Epochs: 1 Epoch (~5,790 steps).
• Final Train Loss: 2.24
• Best Validation Loss: 2.10

📉 Training Loss

Here is the training and validation loss curve during the 1-epoch Supervised Fine-Tuning phase:

🏆 Benchmarks

After the 1-epoch Supervised Fine-Tuning, the text-only model was evaluated on three major benchmarks to test its English comprehension and world knowledge.

Benchmark	Score	What it means
WikiText-2 (Perplexity)	81.49	The model successfully learned standard English grammar, syntax, and punctuation structure. (Lower is better).
SciQ (Accuracy)	35.20%	The model can accurately retrieve basic scientific facts (biology, chemistry) above the 25% random-chance baseline.
MMLU (Accuracy)	23.09%	Expected for this size. The model is too small to memorize college-level law and physics, effectively acting as random chance (~25%).

💻 How to Load and Run

Because this model uses a custom model.py architecture script (included in this repository), you don't load it using the standard transformers library pipeline. Instead, download the files from this repo and use the provided PyTorch script.

import torch
import tiktoken
import json
from model import GPTModel
from safetensors.torch import load_file

# 1. Load Config
with open("config.json") as f:
    cfg = json.load(f)

# 2. Instantiate Model
model = GPTModel(cfg)

# 3. Load Safetensors
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict, strict=False)
model.cuda()
model.eval()

# 4. Tokenizer
tokenizer = tiktoken.get_encoding("gpt2")
eot_token_id = tokenizer.encode("<|endoftext|>", allowed_special={"<|endoftext|>"})[0]

# 5. Inference
prompt = (
    "Below is an instruction that describes a task. "
    "Write a response that appropriately completes the request.\n\n"
    "### Instruction:\nWhat is the capital of France?\n\n### Response:\n"
)

input_ids = torch.tensor(tokenizer.encode(prompt)).unsqueeze(0).cuda()
generated = []

with torch.no_grad():
    for _ in range(100):
        logits = model(input_ids)
        next_token_logits = logits[:, -1, :]
        
        # Repetition Penalty
        penalty = 1.2
        for token_id in set(generated):
            if next_token_logits[0, token_id] < 0:
                next_token_logits[0, token_id] *= penalty
            else:
                next_token_logits[0, token_id] /= penalty
                
        next_token_id = torch.argmax(next_token_logits, dim=-1).unsqueeze(0)
        
        if next_token_id.item() == eot_token_id:
            break
            
        generated.append(next_token_id.item())
        input_ids = torch.cat([input_ids, next_token_id], dim=-1)

print(tokenizer.decode(generated))

📝 Sample Output

When running inference with a repetition penalty of 1.2, the model generates highly coherent text and follows instructions surprisingly well for its size:

Prompt:

How can I stay motivated to exercise?

Output:

1. Set realistic goals and stick to them. This will help you feel more confident in your fitness level, which can lead to better results.
2. Practice mindfulness meditation or yoga regularly. Mindfulness meditation helps reduce stress levels and improve overall well-being.
3. Take breaks throughout the day to recharge and focus on your breath.
4. Exercise regularly. Regular physical activity can help boost energy levels and increase muscle mass.
5. Get enough sleep each night. Sleep is essential for maintaining good health and reducing stress levels.
6. Seek professional advice from a healthcare provider if you have any concerns about your fitness level.