Official Standalone Codebase Notice:
This repository serves as the permanent archival record for the XoneLM-1.0 research paper and baseline model suite.
For the actively maintained, standalone LuminaV optimizer codebase, benchmarks, latest bug-fixes, and ongoing version releases (v1.1.0+), please visit the official upstream repository:
cloverx-id/LuminaV-Optimizer-Paper (Software DOI: 10.57967/hf/10365)

XoneLM & LuminaV Research Papers

(What Happens When You Code at 3 AM: The Architecture & The Optimizer)

Check the 'Files and versions' tab to find the source code.

1. Architecture Paper

XoneLM Architecture (54M)
Read XoneLM.pdf

2. Optimizer Paper

LuminaV Optimizer Theory
Read LuminaV.pdf

Click on each image to read or download its official PDF file.

Research Paper Overview

Abstract

XoneLM is an experimental 54.07M-parameter language model constructed by integrating Stiefel manifold QR decomposition, degree-180 Chebyshev polynomial positional encodings, topological soliton wave tracking, rational power-law attention decay, Poincare hyperbolic routing, and low-rank latent Key-Value compression (MLA).

The entire system was trained from scratch on 32.01M tokens using the custom LuminaV tanh-bounded optimizer on a single consumer GPU in under two hours. The training run achieved monotonic convergence (Final Loss: 1.7355, Perplexity: 5.67) with zero loss spikes, zero gradient explosions, and zero arithmetic underflow errors.

Empirical Telemetry & Training Specs

Metric	Value
Total Active Parameters	54,073,344 (54.07M)
Architecture Backbone	12 Layers, 8 Attention Heads, Dim 512
Key-Value Latent Dimension	64 (Low-Rank Joint MLA)
Working Memory Slots	512 (256 Static + 256 Dynamic)
Optimizer	LuminaV-2B (Learning Rate: 8e-4)
Precision	Mixed Precision FP16
Training Tokens	32,009,639 Tokens (TinyStories)
Wall-Clock Training Time	117.88 minutes (1.96 hours)
Peak Training Throughput	9,868 tokens/second
Peak VRAM Usage	7.29 GB / 14.56 GB (Tesla T4)
Hub Diversity Z-Loss	0.0014
Final Loss / Perplexity	1.7355 / 5.67

Qualitative Analysis: The Box vs. Ball Case Study

During unconditioned zero-shot evaluation on the base pre-trained model:

Prompt: "Once upon a time, Lily found a Box."
Output: "It was a big, round ball. She was so excited to play with it..."

The model produced syntactically perfect English and dialogue quotation marks without infinite looping. However, the pre-training distribution prior of the TinyStories corpus (which heavily features children playing with balls in parks) overrode the prompt keyword. This highlights that unconditioned base models act as probabilistic continuation engines, and Supervised Fine-Tuning (SFT) is necessary for strict instruction adherence.

Architectural Components

Stiefel QR Working Memory Hub: An orthogonal working memory bank initialized via QR factorization on Stiefel manifolds to enforce metric stability from step zero.
PolyHoPE Positional Encodings: Degree-180 Chebyshev polynomials of the first kind evaluated on normalized token intervals to guarantee continuous variance without periodic decay.
Topological Soliton State Tracking: Discrete nonlinear sech-squared wave updates derived from collisionless plasma dynamics to maintain latent memory state stability.
LinHoPE Attention Decay: Heavy-tail Cauchy power-law decay combined with geometric recency bias to mitigate early token context amnesia.
Poincare Hyperbolic Routing: Episodic memory cluster assignment on Riemannian conformal unit disks.
Latent KV Compression: Low-rank Key-Value joint compression (dkv = 64) minimizing memory bandwidth during autoregressive decoding.
LuminaV Optimizer: Master-weight-free parameter optimization featuring a hyperbolic tangent bounding envelope, central innovation variance tracking, and cautious directional masking.

Quickstart: Training and Inference

1. Installation

git clone https://huggingface.co/cloverx-id/XoneLM-1.0-Paper
cd XoneLM-1.0-Paper
pip install -r requirements.txt

2. Running a Training Step with LuminaV

import torch
from tokenizer import build_xonelm_tokenizer
from modeling_xonelm import XoneLM
from luminav import LuminaV

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = build_xonelm_tokenizer()
vocab_size = len(tokenizer)

model = XoneLM(
    vocab_size=vocab_size,
    dim=512,
    num_layers=12,
    num_heads=8,
    kv_latent_dim=64,
    hub_size=512,
    num_terminals=32,
    slots_per_terminal=16
).to(device)

optimizer = LuminaV(
    model.parameters(),
    lr=8e-4,
    betas=(0.9, 0.999),
    tau=0.8,
    buffer=2,
    cautious=True,
    execution="auto"
)

dummy_tokens = torch.randint(0, vocab_size, (2, 512), device=device)
dummy_labels = torch.randint(0, vocab_size, (2, 512), device=device)

optimizer.zero_grad(set_to_none=True)
output = model(dummy_tokens, labels=dummy_labels)
loss = output.loss
loss.backward()
optimizer.step()

print(f"Training step successful. Loss: {loss.item():.4f}")

3. Running Autoregressive Generation (DRY + Min-P)

from generate import generate_response

prompt = "Once upon a time, Lily found a Box."
result = generate_response(
    model=model,
    tokenizer=tokenizer,
    prompt_or_messages=prompt,
    max_new_tokens=64,
    temperature=0.45,
    min_p=0.08
)
print("Output:", result)

Citation

If you use this model, optimizer, or refer to our research, please cite our work as follows:

Primary Citation (Model & Paper)

@misc{luminamoon2026xonelm,
  author       = {{Silver Moon (cloverxion)}},
  organization = {Lumina Moon},
  title        = {{XoneLM: An Over-Engineered 54M Language Model with Non-Euclidean Memory, Polynomial Encodings, and Bounded Optimizers}},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/10270},
  howpublished = {\url{https://huggingface.co/cloverx-id/XoneLM-1.0-Paper}},
  url          = {https://huggingface.co/cloverx-id/XoneLM-1.0-Paper},
  note         = {Hugging Face Model Hub}
}

LuminaV Optimizer (Standalone Codebase & Implementation)

@software{luminamoon2026luminav_code,
  author       = {{Silver Moon (cloverxion)}},
  organization = {Lumina Moon},
  title        = {{LuminaV Optimizer: Official PyTorch Implementation}},
  year         = {2026},
  publisher    = {Hugging Face},
  version      = {1.1.0},
  doi          = {10.57967/hf/10365},
  url          = {https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper}
}

@misc{luminamoon2026luminav_paper,
  author       = {{Silver Moon (cloverxion)}},
  organization = {Lumina Moon},
  title        = {{LuminaV: We Were Too Broke for AdamW So We Trapped Gradients in a Hyperbolic Straitjacket and Hired a Traffic Cop to Slap Them}},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/10270},
  url          = {https://huggingface.co/cloverx-id/XoneLM-1.0-Paper}
}

License

All code, optimizer and architecture assets (including PDFs) are released under the Apache-2.0 License.

Downloads last month: 1,251