sailu4/lmaana-v1

Lmaana v1: Controlled Adaptation for Moroccan Darija ASR

Lmaana v1 is a research checkpoint for Moroccan Darija automatic speech recognition. It continues Lmaana v0 with controlled mixed replay over Dataset13 and MoulSot, aiming to improve the smaller MoulSot domain without erasing performance learned from Dataset13.

TL;DR

• Architecture: OmniASR CTC 1b_v2, trained with fairseq2.
• Method: four controlled sampling pilots followed by one winner continuation.
• Selected mix: 52.64% Dataset13 and 47.36% MoulSot sampling probability.
• MoulSot test WER: 45.2389%, an improvement of 0.1686 point over v0.
• Dataset13 test WER: 44.2228%, a regression of 0.1017 point from v0.
• Checkpoint: native and resumable; this is not a Transformers from_pretrained() export.

1. The Problem: Adaptation Without Forgetting

Moroccan Darija ASR must handle regional vocabulary, code switching, varied orthography, and recording conditions that differ considerably between corpora. Fine-tuning on one domain can improve that domain while damaging another.

This failure appeared clearly before Lmaana v0. Adapting the Dataset13 model only on MoulSot reduced MoulSot WER, but Dataset13 WER deteriorated by almost six points. Lmaana v0 introduced mixed replay and recovered a much stronger cross-domain balance. Lmaana v1 asks a narrower question: can we increase MoulSot exposure further while keeping Dataset13 regression controlled?

Earlier checkpoint	Dataset13 test WER	MoulSot test WER	Observation
Dataset13 step 6,000	44.1165%	46.0435%	Strong original-domain baseline
MoulSot-only step 1,000	50.0524%	43.9947%	Better adaptation, severe forgetting
Lmaana v0	44.1211%	45.4075%	Retained cross-domain baseline

2. Model and Training Setup

Property	Value
Task	Automatic speech recognition
Language	Moroccan Darija (ary)
Architecture	OmniASR CTC 1b_v2 (wav2vec2_asr)
Framework	fairseq2
Tokenizer	Character tokenizer (omniASR_tokenizer_written_v2)
Starting point	Lmaana v0
Optimizer	AdamW, reset at the beginning of each stage
Peak learning rate	1e-7 with a Tri-Stage schedule
Gradient accumulation	8 batches
v1 training budget	500 cumulative updates
Evaluation decoding	Greedy CTC, no external language model

The optimizer was intentionally reset between the selected pilot and its continuation. The pilot learning-rate schedule had already reached its final phase, so restoring it would have produced almost no useful update. The second stage starts from the selected model weights with a fresh optimizer and a new learning-rate cycle, while preserving the original pilot checkpoint.

3. Controlled Sampling Experiments

The pilots changed only beta_corpus, the exponent used to transform corpus hours into sampling probabilities. Lowering this exponent moves the imbalanced corpora toward equal sampling while leaving the model, optimizer, batching, learning rate, and validation schedule unchanged.

Pilot	beta_corpus	Dataset13 weight	MoulSot weight	Purpose
v1-A	0.5	62.91%	37.09%	Conservative continuation
v1-B	0.3	57.86%	42.14%	Increase MoulSot exposure
v1-C	0.2	55.26%	44.74%	Approach balanced replay
v1-D	0.1	52.64%	47.36%	Near-balanced replay

Each pilot ran for 250 updates from the same Lmaana v0 weights. This isolates the effect of corpus sampling and avoids comparing experiments with different training budgets.

4. Validation and Model Selection

The checkpoint selection gate was defined before the final test evaluation:

1. MoulSot validation WER must improve by at least 0.10 point versus v0.
2. Dataset13 validation WER may regress by at most 0.20 point.
3. Among eligible candidates, select the lowest MoulSot WER, using Dataset13 WER as the tie-breaker.

Validation checkpoint	Dataset13 WER	MoulSot WER	Decision
Lmaana v0	43.1870%	40.7053%	Baseline
v1-A	43.2113%	40.7133%	Rejected
v1-B	43.2188%	40.6333%	Rejected
v1-C	43.2091%	40.6183%	Rejected
v1-D, 250 updates	43.2293%	40.5703%	Eligible
v1-D stage 2, 500 cumulative updates	43.2483%	40.5473%	Selected

The final candidate improves MoulSot validation WER by 0.1580 point and regresses Dataset13 by 0.0613 point. It passes both constraints.

Diagnostic Plots

The following screenshots were produced by notebooks/03_results_report.ipynb from the evaluation summaries and fairseq2 logs. They are included as static figures for model-card readers.

MoulSot CTC loss decreases from 76.555 for v0 to 75.500 after the final stage, while Dataset13 remains close to 176.2. The two corpora should be compared across variants rather than directly against each other because their CTC loss scales are corpus-dependent.

Negative values indicate improvement. The selected model improves MoulSot validation WER by 0.1580 point and accepts a controlled 0.0613-point Dataset13 regression.

The training-dynamics screenshot is a diagnostic appendix only: it shows sparse endpoints from separate short runs, not a single continuous convergence curve.

5. Held-Out Test Results

The test partitions were evaluated once, after the final checkpoint had been selected. All results use the same fairseq2 evaluation pipeline and greedy CTC decoding. notebooks/03_results_report.ipynb reconstructs the comparisons as native Matplotlib plots for analysis and report screenshots. The static assets below are maintained separately for the model card.

Model	Test set	CTC loss	UER	WER	Examples
Lmaana v0	Dataset13	183.9310	17.7329%	44.1211%	6,576
Lmaana v0	MoulSot	76.4528	14.8204%	45.4075%	1,960
Lmaana v1	Dataset13	183.9720	17.7658%	44.2228%	6,576
Lmaana v1	MoulSot	75.4045	14.6911%	45.2389%	1,960

Result Analysis

Lmaana v1 improves MoulSot test WER by 0.1686 point while Dataset13 regresses by 0.1017 point. The held-out result therefore confirms the direction seen during validation: near-balanced replay shifts capacity toward MoulSot without reproducing the catastrophic forgetting of MoulSot-only adaptation.

The gain is real but modest. The exploratory MoulSot test target was not reached: WER remained above 45.00%. The much larger gap between UER and WER also suggests that future gains may depend less on additional corpus-beta tuning and more on decoding, word-boundary handling, transcript normalization, and language-model integration.

6. Intended Use

• Research and evaluation of Moroccan Darija speech recognition.
• Continued training with the same OmniASR/fairseq2 stack.
• Development of domain-aware transcription and decoding experiments.
• Comparison of replay strategies for domain adaptation.

This checkpoint has not been validated for high-stakes or fully automated decision-making.

7. Download and Resume

This repository stores the complete native fairseq2/OmniASR checkpoint, including state required for reproducible restoration. It is not a Transformers from_pretrained() export.

from huggingface_hub import snapshot_download

local_path = snapshot_download(repo_id="sailu4/lmaana-v1")
print(local_path)

Load the directory under checkpoint/ with the matching OmniASR fairseq2 recipe. The tokenizer, configuration, selection report, metrics, and evaluation logs are stored under metadata/.

8. Reproducibility and Repository Contents

• checkpoint/: complete native Lmaana v1 checkpoint.
• metadata/: tokenizer, training configuration, reports, and evaluation logs.
• assets/: static figures generated for this model card.
• manifest.json: file sizes and SHA-256 integrity hashes.

The experiment code, configurations, and full protocol are available in the Darija CTC project.

9. Limitations

Performance can vary with recording conditions, speaker demographics, regional vocabulary, code switching, and domains not represented in the evaluation sets. Greedy CTC decoding does not use a language model. The small WER changes have not yet been repeated across multiple random seeds, so statistical uncertainty remains. Review transcriptions before consequential use and review dataset and deployment licenses separately.

Acknowledgements

Lmaana v1 builds on Meta's OmniASR and fairseq2 projects. The checkpoint is published for research on Moroccan Darija ASR and reproducible domain adaptation.

Version

This repository contains Lmaana v1, selected after 500 cumulative updates from the retained Lmaana v0 baseline.