CallEnhancer-Lite — realtime call-centre speech restoration

A realtime-oriented sibling of Scicom-intl/CallEnhancer (enhancer-small). It restores telephony-degraded speech — call-centre channel artifacts, background noise and cross-talk — to clean 24 kHz audio. Same two-part design (a feature predictor followed by a DAC vocoder), but the 580M w2v-BERT 2.0 encoder is replaced by a DistilHuBERT-based encoder (2 transformer layers) plus a small predictor head.

The result is 8.6× fewer parameters and 5.4× the throughput end to end:

	encoder	decoder	total
enhancer-small	w2v-BERT 24L + LoRA — 580M	DAC 3072ch @48 kHz — 188M	768M
enhancer-lite	DistilHuBERT 2L + head — 37M	DAC 1536ch @24 kHz — 52M	89M
enhancer-lite (HiFi-GAN)	DistilHuBERT 2L + head — 37M	HiFi-GAN MRF 1024ch @48 kHz — 61M	98M

Quick start

pip install torch torchaudio "transformers>=4.56" "descript-audio-codec>=1.0.0" soundfile "huggingface_hub[cli]"

hf auth login   # private repo: log in first (or export HF_TOKEN=hf_...)
hf download Scicom-intl/CallEnhancer-Lite \
    infer_callcentre.py expand_decoder.py fe_distilhubert.py \
    fe_lite/fe_only.pt decoder_lite24/decoder_only.pt --local-dir CallEnhancer-Lite

cd CallEnhancer-Lite && python infer_callcentre.py \
    --input your_call.wav --out-dir out \
    --fe-arch distilhubert \
    --fe-adapter fe_lite/fe_only.pt \
    --decoder decoder_lite24/decoder_only.pt \
    --chunk 0 --device cuda   # --chunk 0 = NO chunking (default single pass); --device cpu if no GPU
# -> out/your_call_restored24k.wav (clean 24 kHz)  +  out/your_call_orig24k.wav (A/B)

--input accepts a file or a directory (.wav/.flac/.mp3/.ogg/.opus/.m4a). Stereo (e.g. agent/customer on separate channels) is restored per channel and recombined.

Speed

xRT = seconds of audio per second of compute; higher is better. 2 s chunk, batch 1. The ms columns are CPU (16 threads, fp32); GPU xRT is one H20 (bf16). All rows were measured back to back on the same machine, so they compare like for like; the card was shared at measurement time, so absolute figures are conservative.

stack	encoder	decoder	total	CPU xRT	GPU xRT
enhancer-small (580M + 188M DAC @48 kHz)	202.6 ms	471.9 ms	674.5 ms	3.0	37.0
lite encoder + 52M DAC @24 kHz (this repo)	15.9 ms	107.2 ms	123.1 ms	16.2	123.3
lite encoder + 61M HiFi-GAN @48 kHz	20.0 ms	208.6 ms	228.7 ms	8.7	71.3

The encoder swap alone is a large win on the encoder but a small one end to end — the decoder dominates, so the decoder choice is what sets your throughput.

Files

For inference you only need fe_lite/fe_only.pt + decoder_lite24/decoder_only.pt.

path	role
fe_lite/fe_only.pt	encoder weights (~148 MB) — what inference needs
fe_lite/last.pt	full encoder checkpoint incl. optimizer state (~443 MB)
decoder_lite24/decoder_only.pt	24 kHz decoder weights (~210 MB) — what inference needs
decoder_lite24/last.pt	full 24 kHz decoder checkpoint (~1.1 GB)
decoder_hifigan/decoder_only.pt	48 kHz HiFi-GAN decoder weights (~246 MB) — EARLY snapshot, expect artifacts
decoder_hifigan/last.pt	full HiFi-GAN decoder checkpoint (~1.2 GB)

Python

import torch, fe_distilhubert as fedh
from infer_callcentre import load_decoder

dev  = torch.device("cuda" if torch.cuda.is_available() else "cpu")
fe   = fedh.load_fe("fe_lite/fe_only.pt", dev)
proc = fedh.make_processor()
dec, out_sr = load_decoder("decoder_lite24/decoder_only.pt", dev)

batch = fedh.collate_wavs([wav_16k_float32], proc)          # raw 16 kHz waveform in
feats = fe(**{k: v.to(dev) for k, v in batch.items()}).last_hidden_state   # [1, T, 1024]
wav   = dec(feats.transpose(1, 2)).squeeze().float().cpu().numpy()         # 24 kHz out

Stop after feats if you only want features — they are interchangeable with the 1024-d features the other CallEnhancer decoders take.

Licence

other — provenance is mixed and not uniformly permissive. The upstream model (ntu-spml/distilhubert) is Apache-2.0, but some of the audio data behind these weights carries CC BY-NC and share-alike terms. Review the upstream sources before any commercial use.