maidacundo/open-mythos-hf

OpenMythos-HF — Recurrent Depth Transformer

A clean reimplementation of the OpenMythos architecture in the HuggingFace ecosystem.

What is this?

OpenMythos is a speculative reconstruction of "Claude Mythos" based on several cutting-edge papers. This repo provides a bug-free, HF-native reimplementation that:

1. ✅ Actually runs (fixes tokenizer, dependency, and export bugs)
2. ✅ Integrates with HF Trainer, Trackio, and FSDP
3. ✅ Is fully tested (forward/backward pass, all components)
4. ✅ Trains on real data (FineWeb-Edu)

Architecture

tokens → Embedding → Prelude → [Recurrent Block × r] → Coda → LM Head → logits

Prelude: L_P standard transformer blocks (run once)
Recurrent Block: L_R transformer blocks with state injection, looped r times
Coda: L_C standard transformer blocks (run once)

Key Components

Component	Source Paper	Description
SandwichBlock	Huginn (arXiv:2502.05171)	Pre-norm Attn + Pre-norm MLP + Post-norm for recurrent stability
LTI Injection	Parcae (arXiv:2604.12946)	Stable state mixing with guaranteed ρ(Ā) < 1
Linear Injection	Huginn	Concatenate state + input, project down
Multi-Latent Attention	DeepSeek-V2 (arXiv:2405.04434)	Compressed KV cache (87.5% reduction)
Sparse MoE	DeepSeek-V3	Top-K routed + shared always-active experts
Truncated BPTT	Huginn §3.3	Only backprop through last K recurrence steps
Poisson-Lognormal Sampling	Huginn §3.3	Variable depth per training sample

Model Variants

Variant	Params	n_embd	Recurrence	Extensions
mythos_tiny	12M	256	μ=4	LTI only
mythos_140m	140M	768	μ=8	LTI only
mythos_370m	370M	1024	μ=8	LTI only
mythos_770m	770M	1280	μ=8	LTI only
mythos_1b	~1B	1536	μ=16	LTI + MoE
mythos_3b	~3.5B	2560	μ=32	LTI + MLA + MoE

Quick Start

from open_mythos_hf import OpenMythosForCausalLM, mythos_tiny

# Create a model
config = mythos_tiny()
model = OpenMythosForCausalLM(config)

# Forward pass
import torch
input_ids = torch.randint(0, 32000, (1, 128))
output = model(input_ids=input_ids, labels=input_ids)
print(f"Loss: {output.loss.item():.4f}")
print(f"Recurrence steps: {output.num_steps}")

Training

python train_mythos.py --variant 140m --max_steps 5000 --batch_size 8

What We Fixed vs Original OpenMythos

Bug	Original	Fix
Tokenizer → non-existent model	openai/gpt-oss-20b	Use gpt2 (valid HF tokenizer)
torch = "2.11.0"	Non-existent version	Compatible with PyTorch 2.x
mythos_7b missing	Referenced but not defined	Removed from README
_tied_weights_keys	Wrong format for transformers 5.6+	Dict format
load_tokenizer / get_vocab_size	Exported but undefined	Removed from __all__
No tests	None	Full test suite

References

• Huginn/Raven — Recurrent Depth Transformer
• Parcae — LTI-stable injection
• DeepSeek-V2 — Multi-Latent Attention
• ACT — Adaptive Computation Time
• OpenMythos (original) — Original speculative implementation