simonlesaumon/lrs3-lipreader-visual-only

LRS3 Visual-Only Lip Reading Model

Visual-only English lip-reading model based on the Auto-AVSR architecture (commit 182b628), fine-tuned on the Ainncy/LRS3 trainval split.

Model Description

• Architecture: End-to-end (E2E) visual-only speech recognition, 250M parameters
• Modality: Video only (mouth ROI crops, no audio)
• Frontend: MediaPipe face detection + mouth ROI cropping at 16-second segments
• Training: 20 epochs on a single NVIDIA A100-SXM4-80GB, learning rate 0.001
• Framework: PyTorch 2.5.1 + PyTorch Lightning 2.5.0
• Checkpoint: model_avg_10.pth — average of the last 10 epoch checkpoints

Dataset

Trained on the supervised trainval split from Ainncy/LRS3:

• Training samples: ~98% of trainval (lrs3_train_fit.csv)
• Validation samples: ~2% of trainval (lrs3_train_val.csv)
• Note: The official LRS3 test set is not distributed by Ainncy/LRS3. Validation metrics are computed on a deterministic held-out subset of trainval and should not be compared to published LRS3 test-set results.

Intended Use

This model is intended for research purposes only in visual speech recognition (lip reading). It does not use audio input and is designed to explore the limits of visual-only speech recognition.

Limitations

• Visual-only: Performance is inherently lower than audio-visual or audio-only models.
• English only: Trained solely on English speech from LRS3 (TED/TEDx talks).
• Controlled environment: LRS3 consists of frontal-facing speakers with relatively stable lighting. Real-world performance will be significantly lower.
• No speaker diarization: Cannot distinguish between multiple speakers.
• Vocabulary constrained: Limited to vocabulary present in the training data.
• Privacy & ethics: Lip-reading technology raises privacy concerns. This model should not be used for surveillance or non-consensual speech recognition.

Training Setup

Preprocessing was performed on Modal cloud GPUs using the pipeline in the accompanying repository:

1. Mouth ROI detection with MediaPipe (CPU, 8 parallel shards)
2. Video cropping to 16-second segments
3. Training on 1× A100-80GB for 20 epochs (~4h40)

The raw MP4 video files, crops, and transcriptions are not included in this repository and must not be redistributed without explicit authorization from the dataset copyright holders.

Citation

@article{ma2023auto,
  title={Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels},
  author={Ma, Pingchuan and Haliassos, Alexandros and Fernandez-Lopez, Adriana
          and Chen, Honglie and Petridis, Stavros and Pantic, Maja},
  journal={arXiv preprint arXiv:2303.08807},
  year={2023}
}

License

This model is released under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0), consistent with the LRS3 dataset terms. Commercial use is prohibited.