cagliostro-v1

A 157M parameter decoder-only language model trained from scratch on 36B tokens of web text.

This is a base model. It has not been instruction tuned and will not follow instructions or hold a conversation.

Results

Open SLM Leaderboard Intelligence Index, measured with our own harness:

Benchmark	n	acc	acc_norm
HellaSwag	10,042	31.09 (3,122)	34.41 (3,455)
ARC-Easy	2,376	53.37 (1,268)	46.72 (1,110)
ARC-Challenge	1,172	23.81 (279)	27.82 (326)
Combined ARC		38.59	37.27
PIQA	1,838	65.72 (1,208)	66.70 (1,226)
ArithMark-3.0	1,000	33.60 (336)	33.70 (337)
Intelligence Index		17.85	19.13

The Index is quoted from acc_norm, which is the convention the leaderboard follows. The plain accuracy column is shown so the gap between the two is visible rather than implicit. Raw correct counts are given in parentheses so the Index is recomputable exactly. From those counts it is 19.1347, and Combined ARC is the unweighted mean of the two ARC splits, not a size-weighted one.

BananaMind Base Bench 1.1, scored with the benchmark author's own script:

Category	Elo	Correct	Accuracy	Weighted
language_completion	1570	50/50	100.00%	100.00%
code_completion	1120	24/50	48.00%	53.50%
world_knowledge	1107	37/50	74.00%	70.22%
commonsense	1092	36/50	72.00%	68.45%
logical_reasoning	1002	21/50	42.00%	36.99%
context_tracking	894	18/50	36.00%	35.99%
quantitative	872	14/50	28.00%	27.13%
Overall	1046	200/350	57.14%	53.91%

Both suites were run on the full test sets with no subsampling.

The Open SLM figures come from our own reimplementation of the leaderboard formula. Calibrated against GPT-2, that reimplementation reads about 1.4 points high relative to the published board, so treat 19.13 as our measurement rather than a verified leaderboard result.

The Base Bench figures are not ours. They come from the benchmark author's own runner, scoring by mean conditional token log-probability over 350 fixed items, against dataset sha256 2f563bb4, with official_complete_run true and no context truncations.

Architecture

Parameters	157,411,200
Non-embedding	136,439,680
Layers	30
Hidden size	640
Attention heads	10
Key value heads	5
Head dimension	64
FFN hidden	1,728
Activation	SwiGLU
Normalization	RMSNorm with QK-norm
Position encoding	RoPE, theta 100,000
Vocabulary	32,768
Training context	1,024
Tied embeddings	Yes

The architecture maps exactly onto Qwen3, so the model loads with AutoModelForCausalLM and needs no custom code.

config.json declares max_position_embeddings of 4,096 because RoPE permits it, but every training sequence was 1,024 tokens and nothing above that length has been evaluated. Treat 1,024 as the supported context.

Training

Tokens	36B, single pass, no repetition
Optimizer	AdamW, betas 0.9 and 0.95, weight decay 0.1
Peak learning rate	1.5e-3, cosine decay to 10 percent
Warmup	2,000 steps
Batch	393,216 tokens per optimizer step
Precision	bfloat16
Hardware	8x RTX 5090

Data mixture, held constant throughout with no curriculum:

Source	Share
FineWeb-Edu	57.6%
DCLM-baseline	38.4%
Cosmopedia-v2	4.0%

The mixture was chosen by ablation. Two candidate ratios were trained for 2.5B tokens each and evaluated on the full suite. A FineWeb-Edu heavy mix scored 12.83 and a DCLM heavy mix scored 12.37, so the former was used.

Training data was decontaminated against HellaSwag, ARC, PIQA and ArithMark-3 using a 13-gram overlap filter.

Checkpoint selection

The released weights are from step 91,500 at 36B tokens, not the final step. The model peaked there and declined over the remaining 4B tokens of the cosine tail:

Tokens	Index
5.5B	15.21
10.4B	17.04
15.1B	16.79
19.1B	17.74
23.6B	18.03
27.9B	18.81
32.4B	18.60
36.0B	19.13
39.7B	18.84
40.0B	18.57

A cooldown phase on FineWeb-Edu and Cosmopedia at low learning rate, and weight averaging over the final checkpoints, were both tried on the 40B endpoint. Neither beat the peak checkpoint, reaching 18.94 and 18.80 respectively.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("bench-labs/cagliostro-v1")
tokenizer = AutoTokenizer.from_pretrained("bench-labs/cagliostro-v1")

inputs = tokenizer("The capital of France is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=20, do_sample=False)
print(tokenizer.decode(out[0]))

Limitations

Quantitative reasoning sits near the chance floor at 28 percent. Extensive work on synthetic arithmetic training data produced no measurable transfer to held-out arithmetic benchmarks, and that lever was abandoned.

Context tracking scores 36 percent, the second weakest category. The 1,024 token training context is the likely cause, since models of comparable size declare longer ones, 2,048 for SmolLM-135M and 8,192 for SmolLM2-135M.

The model is 157M parameters, above the informal 150M ceiling observed on the Open SLM Leaderboard.