BDH-150M-EN — Byte-Level BDH Scaling Study Checkpoint

A 150M-parameter BDH (Fast Weight Layers) model trained to completion (30k steps) on a 970M-token byte-level research mix (80% EN fineweb-edu / 20% PL speakleash). This is the largest rung of the SlayerLab BDH scaling ladder.

Training summary

Metric	Value
Params	~151M
Steps	30,000 (completed)
Seq len	2048 (byte-level)
Data	research-mix-v1, 970M tokens, byte-level (vocab 256, no tokenizer)
Optimizer	AdamW lr=1e-3 → 1e-4 cosine, wd 0.1, warmup 1000
Grad clip	ZClip
Final val loss	1.099
Hardware	RunPod A4000, single GPU

Learning curve (val loss per 500 steps, from step 15500)

step   val_loss
15500  1.233
16000  1.325
16500  1.115
17000  1.273
17500  1.460
18000  1.169
18500  1.250
19000  1.258
19500  1.270
20000  1.260
20500  1.203
21000  1.175
21500  1.152
22000  1.241
22500  1.402
23000  1.199
23500  1.125
24000  1.079
24500  1.114
25000  1.215
25500  1.197
26000  1.286
26500  1.178
27000  1.390
27500  1.216
28000  1.531
28500  1.090
29000  1.115
29500  1.186
30000  1.099

Full metrics (incl. steps 0–15000 from the pre-resume backup) in bdh_150M_en.json.

Architecture

BDH from pathwaycom/bdh — looped latent recurrence / fast weights
n_layer=8, n_embd=256, n_head=4, mlp_internal_dim_multiplier=768
Vocab: byte-level (vocab 256) — no tokenizer, works directly on UTF-8 bytes
~151M parameters · seq length 2048

Files

model.safetensors — weights (state dict, model.* keys, compatible with pathwaycom/bdh)
config.json — architecture + training config
bdh_150M_en.json — full training metrics (loss, grad_norm per step)
bdh_150M_en.meta.json — run metadata

Load

import sys, torch
sys.path.insert(0, "path/to/pathwaycom/bdh")
import bdh as bdh_mod
from safetensors.torch import load_file

cfg = bdh_mod.BDHConfig(
    n_layer=8, n_embd=256, n_head=4,
    mlp_internal_dim_multiplier=768, dropout=0.1,
    vocab_size=256,
)
model = bdh_mod.BDH(cfg).bfloat16()
sd = load_file("model.safetensors")
model.load_state_dict({k.replace("model.", ""): v for k, v in sd.items()}, strict=False)

Context

Part of the BDH scaling study (25M → 50M → 150M) — results & baselines comparison in SlayerLab/bdh-scaling. Smaller rung: SlayerLab/bdh-25m-pl.

Downloads last month: 97