yamura4/nelf-s2t-onnx

NeLF S2T ONNX — Flemish Dutch ASR

ONNX export of the NeLF Speech-to-Text PyTorch model. Full Flemish/Dutch ASR with dual verbatim + subtitle output, running via ONNX Runtime — no PyTorch needed.

Model Architecture

• Encoder: STFT → LogMel (80) → GlobalMVN → Conv2dSubsampling6 → 12× Conformer (512-dim, 8 heads)
• Subtitle Encoder: 6× Transformer (cascaded from verbatim encoder)
• Decoders: Dual transformer decoders with swapped cross-attention (verbatim primary / subtitle primary)
• CTC head: Linear projection to 5000 tokens
• Params: 180M

Files

File	Size	Description
encoder.onnx	348 MB	Audio waveform → verbatim encoder features (B, T, 512)
encoder_sub.onnx	83 MB	Verbatim features → subtitle encoder features
ctc.onnx	9.8 MB	Encoder features → CTC logits (B, T, 5000)
decoder_step.onnx	150 MB	Autoregressive decoder step (verbatim)
decoder_sub_step.onnx	150 MB	Autoregressive decoder step (subtitle)
bpe.model	314 KB	SentencePiece BPE tokenizer
tokens.txt	42 KB	Token vocabulary (5000 tokens)

Usage

ONNX Runtime (Python)

import onnxruntime as ort
import numpy as np
import soundfile as sf
from sentencepiece import SentencePieceProcessor

# Load models
enc = ort.InferenceSession("encoder.onnx")
enc_sub = ort.InferenceSession("encoder_sub.onnx")
ctc = ort.InferenceSession("ctc.onnx")

# Load tokenizer
tokenizer = SentencePieceProcessor()
tokenizer.Load("bpe.model")
with open("tokens.txt") as f:
    token_list = [line.strip() for line in f]

# Transcribe
audio, sr = sf.read("speech.wav")
audio = audio.mean(axis=1) if audio.ndim > 1 else audio

wav = audio[np.newaxis, :].astype(np.float32)
wav_len = np.array([wav.shape[1]], dtype=np.int64)

enc_out, enc_len = enc.run(None, {"wav": wav, "wav_len": wav_len})
logits = ctc.run(None, {"enc_out": enc_out})[0]

# Greedy CTC decode
preds = logits[0].argmax(axis=-1)
tokens = []
prev = -1
for p in preds[:int(enc_len[0])]:
    if p != prev and p != 0:
        tokens.append(int(p))
    prev = int(p)

text = tokenizer.DecodePieces([token_list[i] for i in tokens])
print(text)

Autoregressive Decoding

dec = ort.InferenceSession("decoder_step.onnx")

tokens = [1]  # <sos/eos>
for _ in range(225):
    t = np.array([tokens], dtype=np.int64)
    logits = dec.run(None, {
        "tokens": t,
        "encoder_out": enc_out,
        "enc_len": enc_len,
        "sub_out": enc_out,  # or encoder_sub output for subtitle decoder
        "sub_len": enc_len,
    })[0]
    next_t = int(logits[0, -1].argmax())
    if next_t == 1:
        break
    tokens.append(next_t)

Input / Output Shapes

Model	Inputs	Outputs
encoder.onnx	wav: (B, T_wav), wav_len: (B,)	enc_out: (B, T_enc, 512), enc_len: (B,)
encoder_sub.onnx	enc_in: (B, T, 512), enc_len: (B,)	sub_out: (B, T, 512), sub_len: (B,)
ctc.onnx	enc_out: (B, T, 512)	logits: (B, T, 5000)
decoder_step.onnx	tokens: (B, L), encoder_out: (B, T, 512), enc_len: (B,), sub_out: (B, T, 512), sub_len: (B,)	logits: (B, L, 5000)

All axes are dynamic (variable batch size, time, sequence length).

Training Details

Trained on 300 hours of verbatim Flemish CGN (3× noise augmented), 700 hours of Netherlands Dutch CGN, and 14,000 hours of weakly-supervised Flemish broadcast subtitles. See the original model card for full details.

License

CC-BY-NC-4.0 (same as the original NeLF model).