polly-1

A LoRA adapter fine-tuned on top of Qwen/Qwen2.5-0.5B as part of the SCBE-AETHERMOORE governance-and-tongues project. This is a small, fast companion model — the 0.5B footprint is intentional, not aspirational.

What it is

• Adapter type: LoRA (r=16, alpha=32, dropout=0.05)
• Targets: all seven linear projections (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj)
• Base: Qwen/Qwen2.5-0.5B
• Trained with: TRL 1.2.0 + PEFT 0.19.1 on issdandavis/polly-training-data

Training results (2026-04-21)

Trained on Kaggle for 435 steps (3 epochs, effective batch size 16) with cosine LR schedule peaking at 2e-4. Run took ~185 minutes.

Metric	Start	End	Delta
Loss	3.826	1.823	-2.003
Token accuracy	36.82%	67.05%	+30.23pp
Entropy	3.65	1.76	-1.89
Grad norm (max over run)	—	1.672	stable

Loss dropped smoothly without spikes. Gradients stayed under 1.7 throughout — no instability. Final learning rate landed at ~1e-7, meaning the cosine schedule completed cleanly. Entropy dropped alongside loss, which means the model became confidently trained rather than collapsing to a single mode.

Full training report: artifacts/training_reports/polly-1_2026-04-21.json in the SCBE-AETHERMOORE repo.

Quick start

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-0.5B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "issdandavis/polly-1")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")

messages = [{"role": "user", "content": "Explain Sacred Tongues in one sentence."}]
inputs = tokenizer.apply_chat_template(
    messages, return_tensors="pt", add_generation_prompt=True
).to(model.device)

out = model.generate(inputs, max_new_tokens=160, do_sample=True, temperature=0.7)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Training config

base_model: Qwen/Qwen2.5-0.5B
peft: LoRA (r=16, alpha=32, dropout=0.05, bias=none)
target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]
num_train_epochs: 3
per_device_train_batch_size: 4
gradient_accumulation_steps: 4
effective_batch_size: 16
learning_rate: 2e-4
lr_scheduler: cosine
warmup_ratio: 0.03
max_length: 1024
bf16: true
seed: 42

Context

polly-1 is the 0.5B-scale companion adapter in the SCBE-AETHERMOORE project — an AI safety and governance framework built on hyperbolic geometry (Poincare ball model) with a 14-layer security pipeline and six Sacred Tongues (KO, AV, RU, CA, UM, DR) weighted by the golden ratio. See the main repository for architectural details.

Larger siblings in the same family (7B variants, fleet adapters) are separate repos under issdandavis/*.

License

Apache-2.0. Base model license (Qwen2.5-0.5B) applies to merged weights.

Citation

@software{davis2026polly1,
  title  = {polly-1: Sacred Tongues LoRA on Qwen2.5-0.5B},
  author = {Davis, Issac Izreal},
  year   = {2026},
  url    = {https://huggingface.co/issdandavis/polly-1}
}

Trained with TRL and PEFT.