Instructions to use kefir090/Lumen-v2-130M-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use kefir090/Lumen-v2-130M-Base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="kefir090/Lumen-v2-130M-Base", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("kefir090/Lumen-v2-130M-Base", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use kefir090/Lumen-v2-130M-Base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kefir090/Lumen-v2-130M-Base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kefir090/Lumen-v2-130M-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/kefir090/Lumen-v2-130M-Base

SGLang

How to use kefir090/Lumen-v2-130M-Base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "kefir090/Lumen-v2-130M-Base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kefir090/Lumen-v2-130M-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "kefir090/Lumen-v2-130M-Base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kefir090/Lumen-v2-130M-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use kefir090/Lumen-v2-130M-Base with Docker Model Runner:
```
docker model run hf.co/kefir090/Lumen-v2-130M-Base
```

Lumen-v2-130M-Base 💡

Lumen-v2-130M-Base is a high-efficiency Causal Language Model pretrained completely from scratch on a curated 6 Billion token multi-source educational corpus using Google TPU v5e-8.

The model features an 11-layer architecture with tied input/output embeddings, 4 full-rank Value Embedding (ResFormer) memory tables, causal depthwise convolution mixing (Canon K=3), and a single-digit tokenized vocabulary.

📊 Benchmark Results

All evaluations are zero-shot, evaluated with standard length-normalized likelihood (acc_norm) or raw accuracy over the full official test sets. Prompts are evaluated with the prefix <|bos|> token matching the pretraining distribution.

Core Benchmarks

Benchmark	Metric	Score	Samples ($n$)
ArithMark-3	`acc`	69.30%	1,000
PIQA	`acc_norm`	63.33%	1,838
ARC-Easy	`acc_norm`	47.81%	2,376
ARC-Challenge	`acc_norm`	26.37%	1,172
HellaSwag	`acc_norm`	35.21%	10,042
LAMBADA (OpenAI)	`acc`	33.88%	5,153
BLiMP (Grammar)	`acc`	78.90%	67,000
WikiText-2	`bits-per-byte (bpb)`	0.9739	—

Extended Task Breakdown

Task	Evaluation Metric	Score
SciQ	`acc_norm`	77.10%
COPA	`acc`	70.00%
BoolQ	`acc`	56.09%
SWAG	`acc_norm`	51.57%
WinoGrande	`acc`	50.91%
TruthfulQA MC2	`acc`	42.27%
OpenBookQA	`acc_norm`	32.40%
RACE	`acc`	30.05%
CommonsenseQA	`acc`	19.90%
MMLU	`acc`	23.09%
WikiText-2	`word_perplexity`	36.96

🍌 BananaMind Base Bench 1.1

Zero-shot evaluation on the BananaMind Base Bench 1.1 — a 350-example text-completion benchmark for base causal language models. Evaluation uses mean conditional log-probability scoring with no BOS token (official benchmark standard).

Metric	Score
Overall Elo	1022
Raw Accuracy	54.86% (192/350)
Weighted Accuracy	50.86%

Category Breakdown

Category	Elo	Accuracy
Language Completion	1376	96.00% (48/50)
Commonsense	1072	70.00% (35/50)
World Knowledge	1074	70.00% (35/50)
Context Tracking	869	36.00% (18/50)
Quantitative	747	16.00% (8/50)
Logical Reasoning	1021	44.00% (22/50)
Code Completion	1131	52.00% (26/50)

📌 Pretraining Data & Decontamination

The model was pretrained on 5,999,951,872 tokens across a balanced educational blend:

Source	Share	Tokens	Description
`HuggingFaceTB/smollm-corpus` (`fineweb-edu-dedup`)	72.0%	4.32 B	High-quality educational web corpus filtered by classifier score.
`HuggingFaceTB/smollm-corpus` (`cosmopedia-v2`)	10.0%	600 M	Synthetic textbooks, lectures, and articles across broad taxonomies.
`HuggingFaceTB/finemath` (`finemath-4plus`)	7.0%	420 M	Mathematical web pages, derivations, and proofs.
`Procedural Arithmetic`	5.0%	300 M	Algorithmic story problem generator with diverse entities and numbers.
`roneneldan/TinyStories`	3.0%	180 M	Synthetic narratives with constrained vocabulary for syntactic coherence.
`microsoft/orca-math-word-problems-200k`	3.0%	180 M	Grade-school math word problems with step-by-step solutions.

Decontamination Protocol

Full-corpus 13-gram exact-match filtering. All six mix sources were scanned with an Aho-Corasick automaton of 5,288,966 normalized 13-word phrases built from the test and validation splits of every benchmark reported in this card plus GSM8K (HellaSwag, ARC-Easy/Challenge, PIQA, LAMBADA, WikiText-2, SciQ, COPA, BoolQ, SWAG, WinoGrande, TruthfulQA, OpenBookQA, RACE, CommonsenseQA, MMLU, GSM8K, ArithMark-3). 7,590 rows (0.096% of the corpus) were removed across all sources; a post-cut re-scan found zero residual 13-gram overlaps. The released weights were retrained from scratch on the cleaned mix (identical architecture, tokenizer, data order and schedule; the ArithMark-3 generator itself has 0 exact / 0 digit-skeleton / 0 n-gram collisions with the benchmark by construction, and the held-out ArithWord split scores 86.95% vs 69.30% on the benchmark — generalization, not template recall).

🏗️ Architecture & Specifications

Parameter	Value
Architecture	Causal Decoder-Only Transformer (GQA + Tied Weights)
Inference Parameters	127,977,104 (~128M)
Layers	11
Hidden Size (d)	768
Feed-forward Dimension	3072 (ReLU² activation)
Attention Heads	6 Query / 2 KV (GQA 3:1), Head Dim 128
Positional Encoding	RoPE (theta = 100,000)
Normalization	Parameter-free RMSNorm (pre-norm), with per-head QK-norm
Context Window	2048 tokens
Vocabulary Size	32,768 (single-digit byte-level BPE)
Weight Tying	Enabled (wte.weight == lm_head.weight)
Value Embeddings	4 full-rank ResFormer tables on deepest layers
Local Mixing	Causal depthwise 1D convolution (K=3)
Auxiliary Loss	Multi-Token Prediction (MTP t+2, weight 0.3, train-only)
Hardware	Google TPU v5e-8 (128 GB HBM2e)
Training Speed	360,000 tokens/sec (4.5 hours wall-clock time)

💻 Quickstart (Transformers)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "kefir090/Lumen-v2-130M-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# bfloat16 needs compute capability >= 8.0 (Ampere or newer: RTX 30xx+, A100/H100).
# T4/P100 report is_bf16_supported()==True but produce wrong results -- check cc instead.
if torch.cuda.is_available():
    major, _ = torch.cuda.get_device_capability()
    dtype = torch.bfloat16 if major >= 8 else torch.float32
else:
    dtype = torch.float32

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=dtype,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "The solar system consists of"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        top_p=0.85,
        repetition_penalty=1.15,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Transformers version: tested with transformers>=4.57, including the latest 5.x line. Older releases may fail to load the custom architecture correctly.

Note: This is a pretrained base foundation model. It performs autoregressive text completion and is not instruction-tuned.

📜 License

This model and its weights are released under the Apache 2.0 License.

Downloads last month: 872

Safetensors

Model size

0.1B params

Tensor type

BF16