procmarco/pina-500m

Pina 500M

A 536M-parameter decoder-only language model for Italian + English + code, trained from scratch. This is an early / partial base-model checkpoint — about 9.19B tokens (~23% of a planned 40B-token base run) — released for inspection.

This repo contains model weights only (model.safetensors, bf16) plus config and inference code. It intentionally does not include optimizer state; resumable training checkpoints live separately in procmarco/pina-500m-ckpt.

Current source checkpoint: step=17520, token_offset=9186050048.

Current quality

• Italian: grammatical fragments, but still prone to repetition/boilerplate.
• English: improving, but still weak/repetitive.
• Code: still weak; this is not yet a coding-capable model.
• This is a base LM, not instruction-tuned. Use continuation-style prompts.
• Greedy decoding loops badly at this stage; sampling is more informative.

Recent eval notes:

• New representative 16-batch strided eval across the mixed eval bin: 4.466 bits/token, 1.070 bits/byte.
• A broader local 2.10M-token eval gave 3.90 bits/token, 0.967 bits/byte.
• Earlier offset-0 dashboard eval was noisy and not representative of the mixed eval bin.

Model

Custom SparseLM (not a 🤗 Transformers architecture): RMSNorm, RoPE, QK-norm, z-loss, tied embeddings, ReLU-gated sparse FFN, and DeepSeek-style sparse attention (DSA) in warm-up mode (dense flash + lightning-indexer distillation during training). Inference here uses plain dense causal attention.

Config: d_model=1536, n_layers=16, ffn_hidden=4096, n_heads=24, vocab_size=49152, max_seq_len=4096.

Data

40% Italian (FineWeb-2 ita) / 40% English (FineWeb-Edu) / 20% code (github-code-clean).

Tokenizer: procmarco/ita-en-code-bpe-48k. Documents were packed as <|bos|> ... <|eos|>.

Usage

Self-contained — the modeling code (modeling_pina.py) ships in this repo, so you only need torch safetensors tiktoken huggingface_hub:

import json, pickle, torch
from safetensors.torch import load_model
from huggingface_hub import hf_hub_download
from modeling_pina import SparseLM, SparseLMConfig

REPO, TOK = "procmarco/pina-500m", "procmarco/ita-en-code-bpe-48k"
cfg = SparseLMConfig(**{k: v for k, v in json.load(open(hf_hub_download(REPO, "config.json"))).items() if k != "architecture"})
model = SparseLM(cfg).eval()
if torch.cuda.is_available():
    model = model.cuda().to(torch.bfloat16)
load_model(model, hf_hub_download(REPO, "model.safetensors"))

enc = pickle.load(open(hf_hub_download(TOK, "tokenizer.pkl"), "rb"))
bos, eos = enc.encode_single_token("<|bos|>"), enc.encode_single_token("<|eos|>")
ids = [bos] + enc.encode_ordinary("L'Italia è un paese famoso per")
out = model.generate(ids, max_new_tokens=60, temperature=0.8, top_k=40, eos_id=eos)
print(enc.decode(out[1:]))

See example.py for a runnable version.