KOLM-Alpha — Fully-Oscillatory Kuramoto Language Model

Delon Swartz — AI Researcher | Engineer

KOLM-Alpha is, to our knowledge, the first language model with no transformer computation anywhere — both routing between tokens and processing within them are performed by networks of coupled Kuramoto oscillators. It is the successor to KOLM-Hybrid-1, which kept standard attention and made only the feed-forward slot oscillatory. KOLM-Alpha replaces attention itself with synchronization routing.

Naming note: the name KOLM-Alpha previously labelled the hybrid twin study, now published as KOLM-Hybrid-1. Under the project naming registry (2026-07), KOLM-Alpha denotes this fully-oscillatory line (working name "KOLM-True").

The mechanism: KuramotoRouter A

Each layer's routing is a competitive synchronization process, not static attention:

• Softmax competition over the causal past — sharp selection, not blurry averaging.
• RoPE on the routing scores — position is in the routing, not just the moved content.
• Dynamic query modulation — routing weights are recomputed at every settling step, modulated by the evolving oscillator state. This is genuinely dynamic routing that static attention cannot express.

The routed value drives each token's oscillators, which settle via Kuramoto dynamics under trained couplings, natural rotations, and (optionally) frustrated (phase-offset) coupling. Normalization is SphereNorm. Every block is an exact identity at initialization.

The result

Under the same controlled protocol as KOLM-Hybrid-1 (identical tokenizer, data, order, budget, seed; 11.26M TinyStories tokens; single laptop):

Model	Params	Val loss	Perplexity
KOLM-Alpha (Router A)	17.06M	2.3170	10.14
KOLM-Hybrid-1 (attention + osc-FFN)	16.92M	2.6946	14.80
Transformer twin (TMT)	17.54M	2.7188	15.16

KOLM-Alpha wins decisively — 0.38 nats below the hybrid, 0.40 below the transformer twin — while doing zero transformer computation. It crossed the hybrid's final score at 5M tokens (41% through training).

An honest negative precedes this win: Router v1 (sigmoid-gated, position-blind, routing frozen across settle steps) lost at 3.0273. The three diagnosed failures motivated Router A's softmax + RoPE + dynamic modulation.

Scaling to 42M from scratch (the flagship)

KOLM-Alpha-42M ("W42") is the same architecture at 16 layers, d=384, H=320, frustrated coupling, randomized settling depth K ~ U[1,4], trained from scratch on 188M tokens (32k vocabulary, TinyStories + Simple English Wikipedia):

• Val loss 1.7141 / perplexity 5.55 — from scratch, matching function-preserving-transfer-warmed comparables (K-sweep K1 1.7162 / K2 1.6848 / K4 1.7044).
• Chat, via staged SFT: broad chat mixture → single-turn discipline (Alpaca). The staged order beats Alpaca-from-base by 0.35 nats (alpaca-val 3.19 vs 3.55). Turn discipline is clean (answers once, stops at <|end|>); simple facts land ("The capital of France is Paris"). Entity binding and counting remain beyond 42M — capacity limits, not recipe bugs.

Why it matters

The hybrid showed oscillators can process as well as an MLP. KOLM-Alpha shows they can route better than attention — the whole layer is now synchronization dynamics, with a settling-depth compute dial that adds a memory-free axis of scale. Randomized-depth training makes that dial extrapolate past its trained depth.

Files (weights to follow)

File	Purpose
kolm_true.py	full-oscillator model + trainer (Router A)
kuramoto_torch.py	Kuramoto block (parity-tested to 2.2e-16 vs NumPy)
chat_true.py	REPL / sampler with the /think K settling dial
native_true_A.pt	KOLM-Alpha 17M weights (Router A)
native_w42.pt	KOLM-Alpha-42M pretrain weights
native_w42chat_alp.pt	42M chat model (staged SFT)
tiny32k.json	32k tokenizer

Roadmap

• KOLM-Beta-T (repo) — transplanting pretrained transformer weights into this architecture to reach billion-parameter capability without pretraining.
• Larger from-scratch KOLM-Alpha rungs exposing the settling dial as a user-facing "think-harder" control.

Status & limitations

Research preview. Headline comparisons are single-seed; the 17M Router-A run used a settling/lr protocol that differs slightly from the twin runs (a matched rerun is queued before external citation). Corpora are easy and all budgets are far below Chinchilla-optimal; factual recall at 42M is limited.

License

Apache-2.0.

Acknowledgements

Builds on the Kuramoto-oscillator formulation of AKOrN (Miyato et al., ICLR 2025), the frustrated-synchronization principle of FSN (arXiv:2606.18694), and the TinyStories corpus (Eldan & Li, 2023).