Prannesshkva/ISOM-R1-Edge-130M-MoE-Beta-Prototype

ISOM-R1-Edge-130M-MoE: Continuous Recurrent SSM + Mixture-of-Experts

0.0469 MB Invariant State Footprint • Verified on NVIDIA Tesla T4 Cloud GPU • 100K Continuous Token Stream

---

Overview

ISOM-R1-Edge-130M-MoE is an ultra-compact 134.89-million parameter (58.27M active per token) continuous recurrent state-space architecture. Engineered specifically as a high-speed speculative decoding drafter and edge deliberation engine, it processes arbitrarily long token streams without materializing quadratic attention matrices.

---

Model	Primary Architecture Role	Base Lineage (Independent Derivative)	Total / Active Parameters	Max Context	Cache Complexity	Hardware Target
ISOM-R1-Coder-16B-MoE-Beta	160K Bounded Code & MLA MoE	DeepSeek-Coder-V2-Lite (Non-Endorsed)	15.71B / 2.36B Active	163,840 (160K)	O(1) Bounded Manifold (Architectural Spec)	16GB Cloud / Multi-GPU
ISOM-R1-Enterprise-40B-Beta	40B System-2 Foundation Reasoning	Falcon-40B (Non-Endorsed)	40.0B Dense	32,768 (32K)	O(1) Bounded State (Architectural Spec)	Enterprise Multi-GPU (24GB-80GB)
ISOM-R1-Coder-1.5B-Instruct	128K Repository Code Intelligence	Qwen2.5-Coder-1.5B-Instruct (Non-Endorsed)	1.54B Dense	131,072 (128K)	O(1) Bounded State (Tesla T4 Verified)	8GB Developer Laptops / Edge
ISOM-R1-Reasoning-1.5B-Instruct-Beta	32K System-2 Mathematical Deliberation	Qwen2.5-1.5B-Instruct (Non-Endorsed)	1.54B Dense	32,768 (32K)	O(1) Bounded State (Tesla T4 Verified)	8GB Edge / Consumer GPUs
ISOM-R1-Edge-130M-MoE-Beta-Prototype	Unbounded Recurrent Drafter & SSM	Standalone Continuous SSM + MoE	134.89M / 58.27M Active	Unbounded Recurrence	O(1) Recurrent State (0.0469 MB Verified)	Ultra-Low Power Edge & CPU

---

📊 Audited Empirical Hardware Telemetry (NVIDIA Tesla T4, Kaggle Cloud)

Evaluated on an NVIDIA Tesla T4 (14.56 GB / 14,911.7 MB total VRAM, PyTorch 2.10.0+cu128, CUDA 12.8, Kaggle Cloud) across an authentic, unpadded continuous literature stream (Pride and Prejudice, 728,846 characters) from 10,000 up to 100,000 continuous tokens:

Continuous Recurrent Scaling (10,000 to 100,000 Tokens)

Continuous Stream Length	Active Recurrent State	Allocated GPU Memory	Peak GPU VRAM	Generation Throughput	Spatial Complexity Profile	Hardware Status
10,000 tokens	0.0469 MB	1,176.9 MB	2,631.8 MB	8,645.2 tok/s	Constant O(1) (< 1 MB)	SUCCESS
25,000 tokens	0.0469 MB	1,656.4 MB	4,795.1 MB	14,476.6 tok/s	Constant O(1) (< 1 MB)	SUCCESS
50,000 tokens	0.0469 MB	2,614.9 MB	7,686.7 MB	17,844.6 tok/s	Constant O(1) (< 1 MB)	SUCCESS
75,000 tokens	0.0469 MB	2,614.9 MB	8,645.2 MB	19,885.7 tok/s	Constant O(1) (< 1 MB)	SUCCESS
100,000 tokens	0.0469 MB	2,614.9 MB	8,645.2 MB	21,049.0 tok/s	Constant O(1) (< 1 MB)	SUCCESS

Key Architectural Verification: Across the entire 100,000 continuous token stream, the recurrent hidden state footprint remains strictly invariant at 0.0469 MB (48 KB). The continuous Cayley SO(d) manifold preserves numerical isometry, delivering processing speeds scaling up to 21,049.0 tokens/second on a single Tesla T4 GPU.

---

Model Specifications

Parameter	Value
Total Parameters	134.89M Untied (109.16M Tied Backbone) / 58.27M Active per token
Architecture	Continuous Isometric SSM + Mixture-of-Experts
Layers	6 ISOM Recurrent Layers
Hidden Dimension (d_model)	512
SSM State Dimension (d_state)	8
Experts	8 SwiGLU Experts per layer (TopK=2)
Recurrent Working State	0.0469 MB (O(1) constant footprint: 6 layers × 512 channels × 8 states × 2 bytes bf16)
Context Window	Unbounded (constant memory recurrence)
Precision	bfloat16 / float32

---

Quickstart Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "Prannesshkva/ISOM-R1-Edge-130M-MoE-Beta-Prototype"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

prompt = "Solve step by step: If a car travels 90 km/h for 3.5 hours, what is the total distance traveled?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=150,
        temperature=0.7,
        do_sample=True
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

# Optional: Hybrid Neuro-Symbolic Agent Pipeline
from modeling_isom_130m_moe import ISOMDeliberativeAgent
agent = ISOMDeliberativeAgent(model=model, tokenizer=tokenizer)
result = agent.solve("Solve step by step: A train travels 60 km in 45 minutes. What is its speed in km/h?")
print("Agent Trace:", result.get("thought_trace"))
print("Final Answer:", result.get("final_answer"))

---

⚠️ Training Notice: MoE Load-Balancing

ISOMStaticMoE uses softmax top-2 routing with weight renormalization but does not include an auxiliary load-balancing loss. During inference this has no effect — the model routes normally. During fine-tuning, the absence of a balancing term means the router is free to concentrate all tokens on a small subset of experts (expert collapse), leaving the remaining experts with near-zero gradient signal and wasted capacity.

Mitigation for fine-tuning:
Add a Switch Transformer-style auxiliary loss to your training loop before calling loss.backward():

# After computing CE loss, add auxiliary load-balancing loss
router_logits = ...  # collect router_logits from each ISOMStaticMoE layer
probs = torch.softmax(router_logits, dim=-1)                        # [T, E]
f_i = probs.mean(dim=0)                                             # expert fraction
P_i = (probs > 0).float().mean(dim=0)                              # expert dispatch rate
aux_loss = config.num_experts * (f_i * P_i).sum()
total_loss = ce_loss + 0.01 * aux_loss

Without this, or an equivalent expert-parallelism regularizer, training runs longer than ~1,000 steps risk expert utilization collapse to 2 of 8 experts.

---

Citation & Licensing

@software{isom_edge_130m_2026,
  author = {Prannessh K.V.A.},
  title = {ISOM-R1-Edge-130M-MoE: Continuous Recurrent SSM + Mixture-of-Experts Drafter},
  year = {2026},
  publisher = {Zenodo},
  doi = {10.5281/zenodo.22649142},
  url = {https://doi.org/10.5281/zenodo.22649142}
}

• Sole Author & Architect: Prannessh K.V.A.
• LinkedIn: Prannessh K.V.A.
• License: Governed by CC BY-NC-ND 4.0 (Non-Commercial Research) & Enterprise Commercial Terms. See LICENSE.

---

---

Notice of Non-Endorsement & Independent Lineage

Independent Architecture Work: ISOM-R1-Edge-130M-MoE is an original standalone research architecture engineered solely by Prannessh K.V.A. (Author, Architect & IP Holder). It implements continuous isometric state operator manifolds (Cayley SO(d)) combined with 8 SwiGLU Mixture-of-Experts feedforward layers. Governed by CC BY-NC-ND 4.0 & Enterprise Commercial Terms (see LICENSE).