🎙️ NeuroVoice-0.5B (v0.1 Alpha)

NeuroVoice-0.5B (v0.1 Alpha) is an original, autoregressive Neural Audio Language Model built from the ground up for expressive text-to-speech and zero-shot voice cloning. It couples the deep linguistic reasoning of Qwen2.5-0.5B with the high-compression Kyutai Mimi 24 kHz neural audio codec (12.5 Hz frame rate, 8 codebooks).

The model was developed, optimized, and trained on 4x NVIDIA H100 GPUs (DistributedDataParallel) using PyTorch FlashAttention-2 / SDPA, delivering over 99,800 tokens/sec training throughput.

---

🏗️ NeuroVoice Architecture

NeuroVoice operates through a Two-Stage Autoregressive Backbone + Depth Decoder hierarchy:

[Text Prompt + Reference Audio]
             │
             ▼
┌─────────────────────────────────────────────────────────────┐
│ Stage 1: Main Audio Transformer (Qwen2.5-0.5B Backbone)    │
│  - 24 Transformer Layers (d_model=896, 14 Heads, 2 KV Heads)│
│  - SwiGLU MLP (d_ff=4864), RoPE, RMSNorm                    │
│  - Autoregressively predicts Codebook 0 (Semantic & Pitch)  │
└─────────────────────────────────────────────────────────────┘
             │
             ▼  (Audio Hidden States + CB0 Tokens)
┌─────────────────────────────────────────────────────────────┐
│ Stage 2: 4-Layer Depth Decoder (Acoustic Hierarchy)         │
│  - Hierarchically generates Codebooks 1..7                  │
│  - Reconstructs high-frequency acoustics & vocal timbre     │
└─────────────────────────────────────────────────────────────┘
             │
             ▼  (Full 8-Codebook Matrix @ 12.5 Hz)
┌─────────────────────────────────────────────────────────────┐
│ Kyutai Mimi Neural Audio Codec Decoder                      │
│  - Synthesizes 24 kHz Hi-Fi Audio Waveform                  │
└─────────────────────────────────────────────────────────────┘

• Codec Details: Kyutai Mimi compresses 24,000 samples/sec audio into 12.5 frames per second with 8 codebooks (vocabulary size 2048 per codebook), achieving an extreme 1920x temporal compression.
• Stage 1 (Main Backbone): Autoregressively predicts the speech backbone (Codebook 0).
• Stage 2 (Depth Decoder): Takes backbone hidden representations and CB0 to hierarchically generate the remaining 7 acoustic codebooks (CB1 to CB7) with soft acoustic sampling.

---

🍳 Training Recipe & Hyperparameters

Trained on MareNostrum 5 using PyTorch DistributedDataParallel (DDP):

Hyperparameter	Value
Compute Hardware	1 Node / 4x NVIDIA H100 (80GB HBM3)
Dataset	parler-tts/libritts_r_filtered (Clean subset, 21,000 samples, ~35.2 hours)
Global Batch Size	48 (12 per GPU x 4 GPUs)
Optimizer	AdamW ($\beta_1=0.9, \beta_2=0.95, \text{weight_decay}=0.1$)
Learning Rate	$2.0 \times 10^{-4}$ (Cosine Annealing schedule)
Warmup Steps	200 steps (Linear warmup from $1.0 \times 10^{-6}$)
Precision	bfloat16 Mixed Precision (torch.autocast)
Attention Kernel	PyTorch F.scaled_dot_product_attention (FlashAttention-2)
Total Epochs / Steps	15 Epochs / 6,555 Steps
Training Duration	23 minutes 52 seconds
Peak Throughput	~99,800 tokens / second

📉 Loss Trajectory Across Training

Step 0000: Total Loss: 25.1857 | CB0 Loss: 17.3926 | Depth Loss: 7.7931
Step 1000: Total Loss:  3.1959 | CB0 Loss:  0.6863 | Depth Loss: 2.5096
Step 3000: Total Loss:  2.2011 | CB0 Loss:  0.1136 | Depth Loss: 2.0875
Step 5000: Total Loss:  1.4281 | CB0 Loss:  0.0029 | Depth Loss: 1.4252
Step 6555: Total Loss:  1.0407 | CB0 Loss:  0.0034 | Depth Loss: 1.0372 (Best Epoch Avg: 1.1138)

---

🚀 Quickstart & Inference

1. Installation

pip install torch transformers soundfile torchaudio

2. Loading with Hugging Face AutoModel (Recommended)

Thanks to AutoConfig and AutoModel integration (trust_remote_code=True), you can load NeuroVoice directly in Python without cloning the repository:

import torch
from transformers import AutoConfig, AutoModel

repo_id = "TurkishCodeMan/NeuroVoice-0.5B"

# 1. Konfigürasyonu ve Modeli Yükle (Safetensors / bfloat16)
config = AutoConfig.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
model = model.to("cuda" if torch.cuda.is_available() else "cpu")
model.eval()

print(f"✅ {config.model_name} başarıyla yüklendi!")
print(f"Parametre Sayısı: {sum(p.numel() for p in model.parameters()) / 1e6:.2f}M")

---

3. Standalone CLI Inference (Cloned Repository)

You can also clone the repository and use the built-in inference.py script:

git clone https://huggingface.co/TurkishCodeMan/NeuroVoice-0.5B
cd NeuroVoice-0.5B

A. Standard Text-to-Speech

python inference.py \
    --use_qwen_backbone \
    --checkpoint model.safetensors \
    --text "Artificial intelligence is creating the future of speech synthesis." \
    --output output_tts.wav \
    --temperature 0.6 \
    --depth_temperature 0.6 \
    --max_tokens 80

B. Zero-Shot Voice Cloning

Use a 3 to 4-second clean audio sample (.wav) as reference:

python inference.py \
    --use_qwen_backbone \
    --checkpoint model.safetensors \
    --ref_audio reference_voice.wav \
    --ref_max_sec 3.5 \
    --text "Hello! This speech is synthesized using the NeuroVoice neural audio model." \
    --output output_cloned.wav \
    --temperature 0.5 \
    --depth_temperature 0.5 \
    --max_tokens 80

---

🔬 Dataset Analysis & Limitations

The v0.1 model was trained on 35.17 hours of speech from the LibriTTS filtered clean subset:

• Total Samples: 21,000 audio-text pairs
• Average Sentence Duration: 6.03 seconds (18 words, ~4.2 codec frames per word)
• Reference Audio Range in Training: 2.0s to 4.0s (Average 3.71s)

⚠️ Known Limitations in v0.1 Alpha:

1. Audiobook Monotony: LibriTTS is narrated audiobook data; speech has a formal, studio-reader cadence rather than conversational prosody.
2. Optimal Reference Audio: The model was trained with 2-4 second reference clips. Use --ref_max_sec 3.5 for optimal voice cloning quality.
3. Acoustic Timbre: Depth Decoder loss converged to ~1.0, which produces intelligible speech with a slightly synthetic timbre. Scaling up to larger multi-speaker datasets in v0.2 will address this.

---

🗺️ Roadmap (v0.2 & Beyond)

• Scale Training Data: Expand from 35 hours to 500+ hours with diverse conversational audio.
• Enhanced Depth Decoder: Increase Depth Decoder capacity from 4 to 8 layers for warmer human timbre.
• Multilingual NeuroVoice: Extend tokenization to Turkish and multilingual speech.

---

📜 Citation & License

This project is licensed under the Apache 2.0 License.

@misc{neurovoice-0.5b-2026,
  author = {TurkishCodeMan},
  title = {NeuroVoice-0.5B: An Autoregressive Neural Audio Language Model},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/TurkishCodeMan/NeuroVoice-0.5B}}
}