Add model card

README.md

@@ -0,0 +1,90 @@
+---
+language: en
+license: cc-by-nc-4.0
+pipeline_tag: automatic-speech-recognition
+tags:
+  - lip-reading
+  - visual-speech-recognition
+  - auto-avsr
+  - lrs3
+  - audio-visual
+datasets:
+  - Ainncy/LRS3
+metrics:
+  - wer
+model-index:
+  - name: LRS3 Visual-Only Lip Reader
+    results: []
+---
+
+# LRS3 Visual-Only Lip Reading Model
+
+**Visual-only English lip-reading model** based on the [Auto-AVSR](https://github.com/mpc001/auto_avsr) architecture
+(commit `182b628`), fine-tuned on the Ainncy/LRS3 trainval split.
+
+## Model Description
+
+- **Architecture**: End-to-end (E2E) visual-only speech recognition, 250M parameters
+- **Modality**: Video only (mouth ROI crops, no audio)
+- **Frontend**: MediaPipe face detection + mouth ROI cropping at 16-second segments
+- **Training**: 20 epochs on a single NVIDIA A100-SXM4-80GB, learning rate 0.001
+- **Framework**: PyTorch 2.5.1 + PyTorch Lightning 2.5.0
+- **Checkpoint**: `model_avg_10.pth` — average of the last 10 epoch checkpoints
+
+## Dataset
+
+Trained on the **supervised trainval split** from [Ainncy/LRS3](https://huggingface.co/datasets/Ainncy/LRS3):
+
+- **Training samples**: ~98% of trainval (lrs3_train_fit.csv)
+- **Validation samples**: ~2% of trainval (lrs3_train_val.csv)
+- **Note**: The official LRS3 test set is **not** distributed by Ainncy/LRS3.
+  Validation metrics are computed on a deterministic held-out subset of trainval
+  and should **not** be compared to published LRS3 test-set results.
+
+## Intended Use
+
+This model is intended for **research purposes only** in visual speech recognition
+(lip reading). It does **not** use audio input and is designed to explore the
+limits of visual-only speech recognition.
+
+## Limitations
+
+- **Visual-only**: Performance is inherently lower than audio-visual or audio-only models.
+- **English only**: Trained solely on English speech from LRS3 (TED/TEDx talks).
+- **Controlled environment**: LRS3 consists of frontal-facing speakers with relatively
+  stable lighting. Real-world performance will be significantly lower.
+- **No speaker diarization**: Cannot distinguish between multiple speakers.
+- **Vocabulary constrained**: Limited to vocabulary present in the training data.
+- **Privacy & ethics**: Lip-reading technology raises privacy concerns. This model
+  should not be used for surveillance or non-consensual speech recognition.
+
+## Training Setup
+
+Preprocessing was performed on Modal cloud GPUs using the pipeline in the
+[accompanying repository](https://github.com/yoannjardin78/lrs3-lipreader):
+
+1. Mouth ROI detection with MediaPipe (CPU, 8 parallel shards)
+2. Video cropping to 16-second segments
+3. Training on 1× A100-80GB for 20 epochs (~4h40)
+
+The raw MP4 video files, crops, and transcriptions are **not** included in this
+repository and must not be redistributed without explicit authorization from the
+dataset copyright holders.
+
+## Citation
+
+```bibtex
+@article{ma2023auto,
+  title={Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels},
+  author={Ma, Pingchuan and Haliassos, Alexandros and Fernandez-Lopez, Adriana
+          and Chen, Honglie and Petridis, Stavros and Pantic, Maja},
+  journal={arXiv preprint arXiv:2303.08807},
+  year={2023}
+}
+```
+
+## License
+
+This model is released under the Creative Commons Attribution-NonCommercial 4.0
+International License (CC BY-NC 4.0), consistent with the LRS3 dataset terms.
+Commercial use is prohibited.