augustZheng/TTS-Core-AI

TTS Core AI Lab

Local iOS workbench for incrementally porting and profiling the split Kokoro pipeline in Swift.

The app integrates the official KittenTTS Swift package and Supertonic 3 ONNX pipeline, and provides four modes:

• KittenTTS: ONNX Runtime CPU, eight voices, configurable speed.
• Kokoro: the existing 11-stage split Core AI pipeline.
• Supertonic 3: four Core AI models, ten voices, configurable flow steps, 44.1 kHz.
• Compare: sequentially generates all three models for the same text, preserves PCM results for independent playback, and displays wall time, inference time, RTF, Core AI coverage, and speedup.

KittenTTS downloads its Nano model and phonemizer assets on first use. Kokoro and Supertonic 3 use Core AI model assets hosted separately on Hugging Face: https://huggingface.co/augustZheng/TTS-Core-AI.

Model Assets

The Xcode project expects these asset directories:

• TTSCoreAILab/Models
• TTSCoreAILab/SupertonicResources

They are not committed to GitHub because the full asset bundle is about 4.2 GB and contains files larger than GitHub's regular 100 MB file limit.

Download them from Hugging Face before building:

./scripts/download-assets.sh

The script uses uv to run the Hugging Face CLI without creating a project virtual environment.

The current stage runs a fixed Hi. text-to-acoustic-feature chain:

fixed token ids + attention mask
-> ALBERT
-> BERT projection
-> fixed-length duration encoder
-> duration head
-> Swift duration/alignment calculation
-> text encoder CNN + LSTM
-> Swift ASR alignment
-> F0/noise shared LSTM + residual blocks

Bundled assets:

• kokoro_bert_eager_64.aimodel
• kokoro_bert_projection.aimodel
• kokoro_duration_encoder_no_pack_64_intmask.aimodel
• kokoro_duration_head.aimodel
• kokoro_text_encoder_conv_64_intmask.aimodel
• kokoro_text_encoder_lstm.aimodel
• kokoro_f0n_shared_lstm_64.aimodel
• kokoro_f0n_blocks_64.aimodel

How to Try

1. Open TTSCoreAILab.xcodeproj in Xcode.
2. Select the same physical iPhone target that passed the original smoke test.
3. Run the TTSCoreAILab app target.
4. Tap Run Acoustic Stage.

Expected reference result from the Python CoreAI runtime:

• shapes: [1,64,768], [1,512,64], [1,64,640], [1,64,50]
• valid-token durations: [18, 2, 5, 9, 6, 1]
• raw predicted frames: 41
• fixed target frames: 64
• aligned ASR: [1, 512, 64]
• F0: [1, 128]
• noise: [1, 128]

Python CoreAI reference mean-absolute values:

• shared features: 0.35923
• F0: 72.4452
• noise: 6.59444
• text conv: 0.02399
• text hidden: 0.16430
• aligned ASR: 0.18732

The current build now also includes:

• kokoro_decoder_pre.aimodel
• kokoro_generator_core.aimodel
• kokoro_istft.aimodel
• fixed Hi. decoder style and harmonic-source fixtures
• Swift PCM-to-WAV playback through AVAudioPlayer

The current UI mirrors the browser Local Lab:

• editable text and af_heart voice selection;
• selectable bundled Core AI budgets (64 / 64, 128 / 256, and 256 / 512), with automatic profile escalation for longer segments;
• optional automatic segmentation;
• total time, Core AI inference time, audio duration, and RTF;
• per-stage Core AI timing bars;
• generated audio playback controls and a detailed run log.

The PyTorch reference and native Kokoro comparison controls are visible but disabled because those runtimes are not bundled in the iOS app.

The bundled profiles increase the text/frame budget from 64 / 64 to 128 / 256 and 256 / 512; automatic mode chooses the smallest profile that fits a segment. Supertonic ships its validated 64 / 32 baseline assets under SupertonicResources/coreai-assets-baseline.

Tap Generate Locally. A successful run should play generated audio and report:

• generator input: [1, 512, 128]

• spec/phase: [1, 22, 7681]

• audio: [1, 1, 38400]

• audio mean absolute value: approximately 0.01756

• audio min/max: approximately -0.2076 / 0.3555

• RTF Result

The current benchmark on iPhone 17 Pro.

RTF (Real-Time Factor) measures how long it takes to generate one second of audio.

• RTF < 1.0 → Faster than real time (suitable for interactive TTS)

• Lower is better

The benchmark below compares the same prompt on the same device across different runtimes.