lattice-research/lattice-mini

Lattice Mini

A 42M-parameter GPT-2-style decoder-only language model trained from scratch by Lattice. No pretrained weights were used.

Specs

Params	42M
Architecture	GPT-2 (12 layers, 512 embd, 8 heads)
Context	512 tokens
Vocab	8,000 (BPE, byte-level)
Pretraining	WikiText-2, 10k iters
Chat fine-tune	Alpaca-style instruct data (5k examples, 1.2k iters)
Format	### Instruction:\n...\n### Response:\n

Honest expectations

This is a hobby-scale model. It was pretrained on WikiText-2 (a small corpus) and lightly fine-tuned on 5,000 Alpaca examples. It produces repetitive, low-quality text and is not usable for anything serious — it exists to document the training journey and to let people run a real from-scratch model.

Try it in the HF Space or read the write-up on the Lattice blog.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("lattice-research/lattice-mini")
model = AutoModelForCausalLM.from_pretrained("lattice-research/lattice-mini")

prompt = "### Instruction:\nWrite a haiku about trains.\n### Response:\n"
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=60, temperature=0.4)
print(tok.decode(out[0], skip_special_tokens=True))

Weights

• model.safetensors — 42M params, fp32
• Trained with a custom nanoGPT-style implementation; converted to a standard GPT2LMHeadModel (bias-free) so it loads with plain transformers.
• Uses a custom 8k BPE tokenizer (byte-level, <|bos|> <|eos|> <|unk|> <|pad|>).

The Lattice lineup

• Mini (this) — 42M, from scratch
• Spark — 1.5B, Qwen fine-tune, the flagship
• Quark — 1.5B, from scratch, training in progress

More at lattice-research on Hugging Face and lattice-research.vercel.app.