SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer

This repository provides an implementation of SwinLip, the visual speech encoder proposed in:

SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer
Young-Hu Park, Rae-Hong Park, Hyung-Min Park
Neurocomputing, Volume 639, 28 July 2025, 130289
arXiv: 2505.04394

The goal of this codebase is to be a direct, paper-faithful implementation of the SwinLip visual encoder and its evaluation setup. If you are looking for the code that corresponds to the paper, you should be able to rely on this repository alone.

---

Overview

SwinLip replaces the usual ResNet-based visual frontends for lip reading with a lightweight Swin Transformer backed by a Conformer-style temporal module, designed specifically for small mouth ROI inputs (e.g. 96×96 crops from LRW and LRW-1000).

The encoder consists of:

• 3D Spatio-Temporal Embedding Module (Conv3dBNPReLU in modules.py):

  • 3D conv: kernel (3 \times 5 \times 5), stride (1, 1, 1), channels 24
  • BatchNorm3d + PReLU
  • Preserves spatial resolution to support hierarchical Swin stages

• Swin Transformer hierarchy (SwinStage / SwinBlock in modules.py, used in SwinLip in swinlip.py):

  • Initial patch size: 11 × 11
  • Stages 1–3 (Table 1 in the paper):
    • Stage 1: C = 64, depth = 2, window = 4, heads = 2
    • Stage 2: C = 128, depth = 2, window = 4, heads = 4
    • Stage 3: C = 256, depth = 6, window = 2, heads = 8
  • Shifted window self-attention with relative position bias

• 1D Convolutional Attention Module (Conformer1DModule in modules.py):

  • Temporal modeling over encoder outputs as in Eqs. (6)–(9) of the paper
  • Conformer-like block stack:
    • Feed-forward (0.5 scaled residual)
    • 1D multi-head self-attention (optional, removed in streaming mode)
    • Depthwise conv + GLU + pointwise conv (no BN, matching the encoder-side design)
    • Final feed-forward + LayerNorm
  • Supports streaming mode by removing temporal MHSA and BN

Backends used for experiments (Sec. 3, Fig. 2 in the paper) are also provided:

• Word-level backends (LRW / LRW-1000):
  • DCTCNDecoder (decoders.py): DC-TCN-style temporal convolutional network
  • BiGRUDecoder (decoders.py): 2-layer bidirectional GRU backend
• Sentence-level backends (LRS2 / LRS3):
  • ConformerCTCDecoder (decoders.py): Conformer encoder + CTC head

The top-level convenience wrapper SwinLipDecoder composes the SwinLip encoder with one of these backends.

---

Repository structure

• swinlip.py

  • Implements the SwinLip visual encoder (SwinLip), including:
    • 3D Spatio-Temporal Embedding Module
    • Swin Transformer hierarchy
    • 1D Convolutional Attention Module (with streaming option)

• modules.py

  • Building blocks used by SwinLip:
    • Conv3dBNPReLU (3D frontend)
    • WindowAttention, SwinBlock, SwinStage, PatchMerging
    • ConformerConvModule, MHSA1D, Conformer1DBlock, Conformer1DModule

• decoders.py

  • Temporal backends used in the paper:
    • DCTCNDecoder (DCTCN alias) – DC-TCN word-level backend
    • BiGRUDecoder – Bi-GRU word-level backend
    • ConformerCTCDecoder (ConformerCTC alias) – Conformer encoder + CTC backend
    • SwinLipDecoder – convenience wrapper: SwinLip encoder + chosen backend

• data.py

  • Dataset and collate utilities for the benchmarks:
    • LRWDataset, LRWRawDataset – word-level LRW / LRW-1000
    • LRSCTCDataset, LRS2RawCTCDataset, CharTokenizer – sentence-level CTC data for LRS2 / LRS3
    • pad_collate_word, ctc_collate – collation helpers

• preprocess.py

  • MouthROIPreprocessor: MediaPipe-based preprocessing matching Sec. 4.2:
    • Face landmarks → 96×96 mouth ROI
    • Random crop to 88×88 (train) or center crop (eval)
    • Horizontal flip (p = 0.5 on train)
    • Grayscale + normalization

• __init__.py

  • Marks the package; no logic beyond that.

---

Installation

Requirements

This implementation is written in Python and uses PyTorch. A typical environment looks like:

• Python ≥ 3.9
• PyTorch ≥ 2.0 (with CUDA if you want GPU acceleration)
• opencv-python
• mediapipe
• numpy

Install dependencies, for example:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install opencv-python mediapipe numpy

Then install this package in editable mode:

cd path/to/swinlip
pip install -e .

After installation, you can import the components via:

from swinlip.swinlip import SwinLip
from swinlip.decoders import SwinLipDecoder, DCTCNDecoder, BiGRUDecoder, ConformerCTCDecoder
from swinlip.data import LRWRawDataset, LRS2RawCTCDataset

---

Datasets and preprocessing

This repository is structured to match the datasets used in the paper:

• LRW (English word-level) [8]
• LRW-1000 (Mandarin word-level) [24]
• LRS2, LRS3 (English sentence-level) [25, 26]

You must obtain these datasets from their official sources and follow their licenses.

Mouth ROI extraction (LRW / LRW-1000 / LRS2 / LRS3)

For experiments that operate directly on raw videos, use LRWRawDataset / LRS2RawCTCDataset together with MouthROIPreprocessor:

1. Input videos should be in their original dataset folder structure.
2. MouthROIPreprocessor:
   • Detects face landmarks with MediaPipe Face Mesh.
   • Computes the mouth center from lip landmarks.
   • Extracts a 96×96 RGB ROI around the mouth.
   • Converts to grayscale and normalizes to zero mean and unit-ish variance.
   • Applies random 88×88 crops and horizontal flips at training time.

Example usage:

from swinlip.data import LRWRawDataset

train_ds = LRWRawDataset(
    root="/path/to/LRW",
    split="train",
    img_size=88,
)

For sentence-level LRS2:

from swinlip.data import LRS2RawCTCDataset, CharTokenizer

tokenizer = CharTokenizer()
train_ds = LRS2RawCTCDataset(
    root="/path/to/LRS2",
    split="train",
    img_size=88,
    tokenizer=tokenizer,
)

For workflows that use pre-extracted ROIs (e.g., to speed up training), LRWDataset and LRSCTCDataset load tensors of shape [T, 88, 88] from .pt files with accompanying label/transcript JSONs.

---

Using the models

1. SwinLip encoder only

To obtain visual features (before any temporal backend):

import torch
from swinlip.swinlip import SwinLip

encoder = SwinLip(
    img_size=88,
    patch_size=11,
    streaming=False,            # set True for streaming mode
)

batch_size, frames = 4, 29
inputs = torch.randn(batch_size, frames, 88, 88)  # grayscale ROI sequence

features = encoder(inputs)  # [B, T, 512]

This features tensor can be used with any temporal backend.

2. Full SwinLip + DC-TCN (LRW / LRW-1000 word-level)

import torch
from swinlip.swinlip import SwinLip
from swinlip.decoders import SwinLipDecoder

num_classes = 500  # LRW; use 1000 for LRW-1000

encoder = SwinLip(img_size=88, patch_size=11)
model = SwinLipDecoder(
    encoder=encoder,
    decoder_type="dctcn",
    in_dim=512,
    num_classes=num_classes,
)

inputs = torch.randn(8, 29, 88, 88)  # [B, T, H, W]
logits = model(inputs)               # [B, num_classes]

3. SwinLip + Bi-GRU (LRW-1000 word-level)

from swinlip.swinlip import SwinLip
from swinlip.decoders import SwinLipDecoder

encoder = SwinLip(img_size=88, patch_size=11)
model = SwinLipDecoder(
    encoder=encoder,
    decoder_type="bigru",
    in_dim=512,
    num_classes=1000,
    hidden_size=512,
    num_layers=2,
)

4. SwinLip + Conformer CTC (LRS2 / LRS3 sentence-level)

import torch
from swinlip.swinlip import SwinLip
from swinlip.decoders import SwinLipDecoder

vocab_size = 40  # depends on tokenizer

encoder = SwinLip(img_size=88, patch_size=11)
model = SwinLipDecoder(
    encoder=encoder,
    decoder_type="conformer",
    in_dim=512,
    vocab_size=vocab_size,
    num_layers=4,
    heads=8,
)

inputs = torch.randn(2, 75, 88, 88)       # [B, T, H, W]
logits = model(inputs)                    # [B, T, vocab_size] for CTC

You would then compute the cross-entropy loss (LRW/LRW-1000) or CTC loss (LRS2/LRS3) on top of these logits.

---

Configuration and defaults (Table 1)

By default, SwinLip is instantiated with parameters that match Table 1 of the paper:

• 3D frontend:
  • Conv3D (3 \times 5 \times 5), stride (1, 1, 1), C = 24
• Swin hierarchy:
  • Patch partition: C = 64, P = 11
  • Stage 1: C = 64, P = 2, N = 2, window = 4, heads = 2
  • Stage 2: C = 128, P = 2, N = 2, window = 4, heads = 4
  • Stage 3: C = 256, P = 2, N = 6, window = 2, heads = 8
• Temporal module:
  • Stage 4: temporal dimension 512, N = 2, heads = 16

These defaults are exposed via the constructor arguments in SwinLip:

• img_size=88
• patch_size=11
• base_dim=64
• depths=(2, 2, 6)
• window_sizes=(4, 4, 2)
• heads=(2, 4, 8)
• temporal_dim=512
• temporal_depth=2
• temporal_heads=16
• streaming=False by default, True to remove MHSA in the temporal module.

---

Reproducing results (high-level)

The exact training schedules (learning rates, warmup, schedulers) for each dataset are described in Sec. 4.3 of the paper. This repository provides the model architectures and data preprocessing faithfully; you can plug these into your own training scripts following the hyperparameters from the paper:

• Word-level (LRW / LRW-1000):

  • Loss: cross-entropy
  • Backend: DC-TCN or Bi-GRU
  • Input: grayscale mouth ROIs, 88×88, 29 (LRW) or 40 (LRW-1000) frames

• Sentence-level (LRS2 / LRS3):

  • Loss: CTC over character sequences
  • Backend: Conformer encoder + CTC
  • Input: grayscale mouth ROIs, variable length

You can use torch.utils.data.DataLoader together with the datasets in data.py and standard PyTorch training loops.

---

Citation

If you use this implementation in your work, please cite the original SwinLip paper:

@article{park2025swinlip,
  title   = {SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer},
  author  = {Park, Young-Hu and Park, Rae-Hong and Park, Hyung-Min},
  journal = {Neurocomputing},
  volume  = {639},
  pages   = {130289},
  year    = {2025},
  doi     = {10.1016/j.neucom.2025.130289},
  archivePrefix = {arXiv},
  eprint  = {2505.04394},
}

For the referenced backends and datasets, please also cite the corresponding works (LRW, LRW-1000, DC-TCN, Bi-GRU, Conformer, etc.) as listed in the SwinLip paper [2505.04394].

---

License

See LICENSE for details.