SlayerLab/gollem-v4-250m-pl

GoLLeM v4 — 250M (Polish, research checkpoints — training trajectory)

⚠️ STATUS: RESEARCH / TRAINING IN PROGRESS. This repository contains intermediate training checkpoints (a series of snapshots covering roughly 4–82% of the planned training budget), released for research purposes only. This is not a finished model. Quality improves with every snapshot and will be substantially higher at the end of training. Do not use it in production, and do not treat these checkpoints as a measure of "how good GoLLeM is".

GoLLeM v4 is a 250M-parameter decoder-only (GPT-style) Polish language model, trained from scratch on a Polish corpus. It is a Slayer Lab project. We publish a series of snapshots from successive stages of training so that others can study how the model learns over time.

Author: Arkadiusz Słota (Slayer Lab).

What it is for (and what it is not)

Intended use: research on scaling and training dynamics (comparing successive snapshots), analysis of a Polish language model, and tests of fluency and text continuation in Polish.

Not an instruction-following model. This is a base (pretrained) model, not an instruction-tuned one: it continues the text you give it, it does not follow commands. It understands Polish only — by design, it performs poorly in English and when asked questions without context. At this stage it does not reliably end its output (it does not consistently emit the <|endoftext|> token) and will generate text up to the length limit. This is normal for a base model early in training, not a bug.

Training snapshots (trajectory)

Snapshot	Step	% of training	Validation loss	BPB
ckpt_2000	2000	~4%	3.416	1.135
ckpt_6000	6000	~12%	2.966	1.001
ckpt_8000	8000	~16%	2.937	0.978
ckpt_10000	10000	~20%	2.878	0.961
ckpt_12000	12000	~25%	2.846	0.947
ckpt_14000	14000	~29%	2.849	0.938
ckpt_16000	16000	~33%	2.789	0.932
ckpt_18000	18000	~37%	2.799	0.924
ckpt_20000	20000	~41%	2.757	0.915
ckpt_22000	22000	~45%	2.756	0.910
ckpt_24000	24000	~49%	2.689	0.903
ckpt_26000	26000	~53%	2.670	0.901
ckpt_28000	28000	~57%	2.652	0.897
ckpt_30000	30000	~61%	2.672	0.891
ckpt_32000	32000	~66%	2.692	0.888
ckpt_34000	34000	~70%	2.686	0.884
ckpt_36000	36000	~74%	2.639	0.881
ckpt_38000	38000	~78%	2.650	0.877
ckpt_40000	40000	~82%	2.643	0.875

New snapshots are added as training progresses (currently up to ckpt_40000, ~82% of the budget).

The model is genuinely learning. BPB falls monotonically (0.978 → 0.875), and validation loss tracks training loss closely, so there is no sign of overfitting. The small bumps in validation loss are single-batch noise, not a change in trend.

How we validate the forecast. We test the power-law forecast out of sample, on steps that were not part of the fit:

• The prediction for step 18000 was on target.
• Between steps 20000 and 26000 the model came in below the extrapolation — it learned faster than the power law predicted.
• Between steps 26000 and 40000 results match the refreshed fit within Δ ≈ −0.002 to −0.004 BPB. The forward check has now held eight times in a row (steps 26000–40000), including every check through the learning-rate annealing phase that began at step 30000. At step 34000: measured 0.884 vs. predicted 0.8875 (Δ = −0.0035); at step 40000: measured 0.875 vs. predicted 0.878 (Δ = −0.003), both below 3× RMS.
• An anti-collapse diversity monitor (distinct-n on generated text) flagged a transient dip at step 38000 (distinct-3 fell to 0.930). This was a benign annealing transient, not memorization: the direct held-out test excluded memorization — held-out BPB kept falling through the flagged window (0.8807 → 0.8772 → 0.8746) — and distinct-n rebounded to 0.961 at step 40000. Snapshots 34000 / 36000 / 40000 are confirmed benign.

End-of-training forecast (24B tokens, ~48,828 steps): BPB around 0.87–0.89. This is a rough range, not a formal confidence interval, from a power-law fit to 11 data points (steps 2000–22000), later refreshed. Treat it as provisional and not yet locked in: the original fit used points from before learning-rate annealing, and the curve can change slope during annealing. The first tail-phase point (step 40000) now holds the refit (measured BPB 0.875 vs. predicted 0.878, within 3× RMS), so the endpoint estimate is firming (currently around 0.869) — but this is not yet endpoint-confirmed; confirmation needs sustained points beyond step 40000. The theoretical floor for the language is around 0.83 BPB. We report the forecast as an estimate with uncertainty, not a hard number. Expected validation loss: ~2.5–2.8 (a noisy metric with an uncertain final value; with annealing, the real result is likely closer to the lower end). So far the model beats the power-law forecast on two independent metrics (validation loss and BPB) — the improvement is real, and the ranges will narrow as training continues.

Architecture

Parameters	247,954,432 (~250M, with tied embeddings)
Layers (n_layer)	17
Model width (d_model)	1024
Attention heads (n_head)	16 (head dim 64)
Context length (block)	1024
Vocabulary	32,000 (V32k tokenizer, Polish BPE)
Positional encoding	learned positional embeddings (nn.Embedding, not RoPE)
Block	pre-LayerNorm, causal SDPA attention, 4× GELU MLP
Weights	input embedding and output head are tied (weight tying, as in GPT-2)

The architecture is custom and does not load via transformers/AutoModel. Use the included loader modeling_gollem.py (standalone; requires only torch, tokenizers and safetensors).

Training data

Corpus: SlayerLab/gollem-corpus-16b-pl — 16.58B tokens, 100% Polish, cleaned (deduplication, PII filtering, decontamination):

Register	Tokens	Share
web (HPLT v3 PL, cleaned)	14.62B	88.2%
encyclopedic (Polish Wikipedia)	984M	5.9%
science	522M	3.1%
news	211M	1.3%
legal	176M	1.1%
literary	60M	0.4%
mixed	1.6M	0.01%

Training target: 24B tokens (~1.45 epochs). This is deliberate overtraining — roughly 5× the Chinchilla-optimal token count — which is optimal for the cost of later inference rather than for training itself. Hardware: a single RTX 5090.

Usage

pip install torch tokenizers safetensors
# latest snapshot (default):
python generate.py "Polska to kraj" --n 100 --temp 0.8 --topk 50
# a specific snapshot from the trajectory:
python generate.py "Polska to kraj" --ckpt ckpt_6000.safetensors --n 100

from modeling_gollem import GollemGPT
m = GollemGPT.from_pretrained("./")                          # latest snapshot
# m = GollemGPT.from_pretrained("./", ckpt="ckpt_6000.safetensors")  # a specific stage
print(m.generate("Stolica Polski to", max_new_tokens=100, temperature=0.8, top_k=50))

Files in the repository: ckpt_*.safetensors (weights of each snapshot, pickle-free format), config.json (architecture and list of snapshots), tokenizer.json (V32k), modeling_gollem.py and generate.py (standalone loader and generation script).

Note on format: weights are stored as safetensors (pure tensors, no executable code), which is safer for public release than raw .pt (pickle) files. Full training checkpoints (with optimizer state, for resuming training) remain internal.

Limitations and ethical notes

• These models are intermediate and immature and do not reflect final quality.
• They were trained on web data; despite filtering, they may reproduce biases, errors and content present in the corpus.
• A PII filter was applied (residual rate ≈ 0.0018%), but this does not guarantee the complete absence of personal data.
• Research use only. Slayer Lab accepts no responsibility for production use.

License and data provenance

The data comes from, among other sources, the HPLT v3 (PL) corpus and CC-BY-SA resources (the encyclopedic register and parts of the legal and literary registers). The provenance of each register is documented in the corpus manifest. The weights are released for research purposes.

---

GoLLeM v4 — Slayer Lab. Author: Arkadiusz Słota. Research checkpoints (training trajectory), work in progress; card maintained by the Slayer team.