| ---
|
| license: mit
|
| language:
|
| - en
|
| library_name: cflow
|
| tags:
|
| - moe
|
| - cpu-inference
|
| - rust
|
| - custom-architecture
|
| - pipeline-native
|
| - avx-512
|
| datasets:
|
| - roneneldan/TinyStories
|
| - HuggingFaceFW/fineweb-edu
|
| pipeline_tag: text-generation
|
| model-index:
|
| - name: arch2_4_combined
|
| results:
|
| - task:
|
| type: text-generation
|
| dataset:
|
| name: TinyStories
|
| type: roneneldan/TinyStories
|
| metrics:
|
| - name: Test Perplexity (114M, 10K steps)
|
| type: perplexity
|
| value: 6.50
|
| - name: Top-1 Accuracy (114M, 10K steps)
|
| type: accuracy
|
| value: 56.8
|
| - name: Val Perplexity (8.34B / 4-layer, 10K steps)
|
| type: perplexity
|
| value: 4.52
|
| - name: Top-1 Accuracy (8.34B / 4-layer, 10K steps)
|
| type: accuracy
|
| value: 61.4
|
| ---
|
|
|
| # arch2_4_combined — Pipeline-Native MoE for CPU Inference
|
|
|
| A custom decoder-only transformer with delayed dense FFN + delayed MoE experts,
|
| designed so its inter-layer dependency graph permits vertical pipelining on CPU.
|
| Part of the **cflow** project — a CPU-first streaming inference engine written in
|
| Rust.
|
|
|
| > **Hosted weights:** this repository hosts `model.cflow` (17.39 GB) — the
|
| > **arch2_4_8k_16l** model: 16 layers, hidden 8192, **~31B parameters**
|
| > (top-2-of-8 MoE, ~20B active/token), Q4. This is the model benchmarked at
|
| > 5.94 tok/s below. The **8.34B** figures in this card refer to a *smaller
|
| > 4-layer scale point* (`arch2_4_8k_4l`) used for quality and cache-locality
|
| > validation (val ppl 4.52); that checkpoint is not hosted here.
|
|
|
| ## Key Results
|
|
|
| | Metric | Value |
|
| |---|---|
|
| | CPU decode throughput (~31B / 16-layer, Q4, 32 threads) | **5.94 tok/s** |
|
| | Effective memory bandwidth | 61 GB/s (30% of 204.8 GB/s peak) |
|
| | Bandwidth reduction from pipelining | **2.00x** (9.00 → 4.50 MB/token) |
|
| | Test perplexity (114M, TinyStories, 10K steps) | 6.50 |
|
| | Val perplexity (8.34B / 4-layer, TinyStories, 10K steps) | 4.52 |
|
|
|
| ### CPU Decode Benchmark (AWS r6i.8xlarge, Ice Lake Xeon, 256 GB DDR4)
|
|
|
| | Engine | Model | Quant | tok/s |
|
| |---|---|---|---|
|
| | **cflow** | arch2_4_8k_16l (~31B MoE, ~20B active) | Q4 | **5.94** |
|
| | Ollama (llama.cpp) | Qwen2.5-32B (32B dense) | Q4 GGUF | 4.75 |
|
| | vLLM CPU | Qwen2.5-32B-Instruct (32B dense) | GPTQ-Int4 | 1.65 |
|
|
|
| > **Note:** cflow and the baselines run different models — cflow's ~31B MoE has
|
| > ~20B active params per token vs 32B dense. The total parameter counts are
|
| > comparable (31B vs 32B), but the architectures and training differ, so the
|
| > cflow number shows what a co-designed architecture + streaming runtime achieves,
|
| > not a quality-matched result.
|
|
|
| ## Model Description
|
|
|
| **arch2_4_combined** is a pre-norm decoder-only transformer with a parallel dense
|
| FFN + sparse MoE block per layer, using delayed residual injection:
|
|
|
| - The **dense FFN** reads from a delayed residual (1 layer behind)
|
| - The **MoE experts** are routed on the current residual but injected 2 layers later
|
| - This creates a dependency DAG where dense and expert weight reads for layer N
|
| can overlap with compute for layer N-1, reducing critical-path memory bandwidth
|
|
|
| The architecture was selected from a screen of 5 pipeline-native candidates. It
|
| is the only design that achieves a measured bandwidth reduction (2.00x) while
|
| maintaining competitive perplexity.
|
|
|
| ### Architecture Details
|
|
|
| | Parameter | 114M (screening) | ~31B (16-layer, hosted) |
|
| |---|---|---|
|
| | Hidden dim | 512 | 8,192 |
|
| | Layers | 6 | 16 |
|
| | Attention heads | 8 | 128 |
|
| | Head dim | 64 | 64 |
|
| | Dense FFN hidden | 2,048 | 32,768 |
|
| | Expert FFN hidden | 512 | 4,096 |
|
| | Experts / top-k | 8 / 2 | 8 / 2 |
|
| | Dense delay | 1 | 1 |
|
| | Expert delay | 2 | 2 |
|
| | Vocab | 50,257 (GPT-2 BPE) | 50,257 (GPT-2 BPE) |
|
| | Max seq len | 512 | 2,048 |
|
|
|
| ### Per-Layer Forward Pass
|
|
|
| ```
|
| attn_out = attention(attn_norm(x))
|
| x = x + attn_out # residual connection
|
| x = x + dense_ffn(ffn_norm(delayed_x)) # dense reads DELAYED residual
|
| if queued_expert: x = x + queued_expert # inject expert from 2 layers ago
|
| expert_out = moe(ffn_norm(x)) # router sees CURRENT residual
|
| # expert_out queued for injection at layer + expert_delay
|
| ```
|
|
|
| ### Components
|
|
|
| - **Attention:** Multi-head (not GQA), Q/K/V/O projections (no bias), standard
|
| RoPE (base=10000, half-interleave), causal masking, KV cache
|
| - **Dense FFN:** GeGLU — `down(gelu(gate(x)) * up(x))`
|
| - **MoE:** Linear router → top-k selection → softmax over selected → per-expert
|
| GeGLU FFN → weighted sum. No auxiliary/load-balancing loss.
|
| - **Normalization:** RMSNorm (eps=1e-6) at attn input, FFN input, and pre-lm_head
|
| - **Combine style:** `DelayedSum` — dense and router share `ffn_norm` but read
|
| different residual snapshots
|
|
|
| ## Training
|
|
|
| ### 114M Screening (5 architectures)
|
|
|
| | | |
|
| |---|---|
|
| | Dataset | TinyStories (431M train tokens, 24M test tokens) |
|
| | Tokenizer | GPT-2 BPE (50,257 vocab) |
|
| | Sequence length | 512 |
|
| | Optimizer | AdamW (betas=0.9/0.95, eps=1e-8, weight_decay=0.1) |
|
| | Learning rate | 3e-4 with linear warmup (200 steps) + cosine decay to 1e-5 |
|
| | Gradient clipping | Global norm 1.0 |
|
| | Batch size | 8 |
|
| | Steps | 10,000 |
|
| | Precision | float32 |
|
| | Hardware | RTX 3060 12 GB |
|
|
|
| ### 8.34B Scale-Up (4-layer — quality & cache validation)
|
|
|
| This is the smaller scale point: `arch2_4_8k_4l`, 4 layers, 8.34B params. It
|
| provides the quality numbers (val ppl 4.52, top-1 61.4%) and the PMU cache-locality
|
| result. The hosted decode-benchmark model (`arch2_4_8k_16l`, ~31B) shares this
|
| per-layer geometry but has 16 layers.
|
|
|
| | | |
|
| |---|---|
|
| | Dataset | TinyStories (same splits) |
|
| | Optimizer | 8-bit AdamW (bitsandbytes) |
|
| | Learning rate | 1e-4 with linear warmup (500 steps) + cosine decay to 1e-6 |
|
| | Batch size | 4 per GPU (global 32) |
|
| | Steps | 10,000 |
|
| | Precision | bf16 |
|
| | Parallelism | FSDP (FULL_SHARD / ZeRO-3) |
|
| | Gradient checkpointing | Per `DelayedMoELayer`, non-reentrant |
|
| | Hardware | 8x A100 SXM4 80 GB (Lambda Cloud) |
|
|
|
| ### Architecture Comparison (114M, TinyStories, 10K steps)
|
|
|
| | Architecture | dense_delay | expert_delay | Test PPL | Top-1 Acc | BW Reduction |
|
| |---|---|---|---|---|---|
|
| | arch1_decoupled_streams | 0 | 0 | 7.21 | 54.9% | 1.00x |
|
| | **arch2_4_combined** | **1** | **2** | **6.50** | **56.8%** | **2.00x** |
|
| | arch3_pipeline_registers | 0 | 0 | 7.24 | 55.1% | 1.00x |
|
| | arch4_async_experts | 0 | 2 | **6.26** | **57.6%** | 1.00x |
|
| | arch5_fixed_point | 0 | 0 | 6.77 | 56.2% | 1.00x |
|
|
|
| **Key insight:** Dense delay is the bandwidth knob; expert delay is the quality
|
| knob. arch4_async_experts gets the best perplexity by routing off pre-dense
|
| activations (cleaner router signal) but sacrifices the bandwidth win that
|
| arch2_4 achieves by also delaying the dense read.
|
|
|
| ## Inference with cflow
|
|
|
| cflow is a Rust inference engine that reads `.cflow` (per-layer streaming) or
|
| `.vflow` (vertical pipeline) weight files. Weights are stored as pre-tiled Q4
|
| (128x256 tiles, ~18 KB each, sized to fit L2 cache).
|
|
|
| ```bash
|
| # Build
|
| cargo build --release --bin cflow-run
|
|
|
| # Convert safetensors → .cflow
|
| cargo run --release --bin cflow-convert -- \
|
| --input checkpoint.safetensors \
|
| --output model.cflow \
|
| --model arch2_4
|
|
|
| # Run inference
|
| CFLOW_THREADS=32 ./target/release/cflow-run \
|
| model.cflow 32 \
|
| --prompt "Once upon a time" \
|
| --tokenizer tokenizer.json \
|
| --temperature 0.8
|
| ```
|
|
|
| ### SIMD Support
|
|
|
| The runtime auto-detects and dispatches to the best available instruction set:
|
|
|
| | ISA | Kernel | Notes |
|
| |---|---|---|
|
| | AVX-512 + VNNI | Q4×Q8 `vpdpbusd` | Best path (Ice Lake+) |
|
| | AVX-512F | Q4×f32 FMA | Skylake-X+ |
|
| | AVX2 + FMA | Q4×f32 FMA | Haswell+ |
|
| | AVX + SSE4.1 | Q4×f32 | Sandy Bridge+ |
|
| | Scalar | Q4×f32 | Fallback |
|
|
|
| ## Limitations
|
|
|
| - **Not a general-purpose LLM.** Trained on TinyStories / FineWeb-Edu subsets at
|
| 10K steps — this is an architecture and runtime research artifact, not a
|
| production language model.
|
| - **Custom architecture.** Cannot be loaded in Hugging Face Transformers, vLLM,
|
| or llama.cpp without adaptation. Requires the cflow Rust runtime or the
|
| PyTorch reference in `pipeline_native/`.
|
| - **CPU-only.** The runtime targets x86-64 CPUs with AVX2 or AVX-512. No GPU
|
| backend.
|
| - **Single-token decode optimized.** Batch/prefill throughput is not the focus.
|
|
|
| ## Thesis Scorecard
|
|
|
| The cflow project tests 8 claims about CPU inference optimization:
|
|
|
| | # | Claim | Result |
|
| |---|---|---|
|
| | 1 | Conditional expert reading (top-k only) | **Proven** |
|
| | 2 | Tile-streaming L1/L2 cache locality | **Proven** (7.29x fewer L1-d misses, PMU-measured) |
|
| | 3 | AVX2/AVX-512 Q4 SIMD kernels | **Proven** |
|
| | 4 | Fused QKV and gate+up projections | **Proven** |
|
| | 5 | Compute-order file layout | **Proven** |
|
| | 6 | Software prefetch (`_mm_prefetch`) | **Refuted** (no benefit; slightly harmful) |
|
| | 7 | Vertical pipeline via delayed dependencies | **Validated** (2.00x bandwidth reduction) |
|
| | 8 | Stage-major disk layout readahead | **Inconclusive** (no isolated benefit; test confounded) |
|
|
|
| ## Citation
|
|
|
| ```bibtex
|
| @software{poperszky2026cflow,
|
| author = {Poperszky, Tom},
|
| title = {cflow: CPU-First Streaming Inference for Pipeline-Native Transformers},
|
| year = {2026}
|
| }
|
| ```
|
|
|
| ## License
|
|
|
| MIT
|
|
|
