TalTech ASR

FastAPI server for Automatic Speech Recognition with Estonian language optimization, speaker diarization, and distributed processing.

Features

• Estonian Speech Recognition using TalTechNLP fine-tuned Whisper model
• Speaker Diarization with pyannote.audio for multi-speaker identification
• Language Detection with automatic model selection (Estonian vs. generic)
• Multiple Output Formats (JSON, SRT, TXT, WebVTT)
• Distributed Processing with Ray for scalable parallel execution
• Optional Overlap Handling with MossFormer2 speech separation (disabled by default)
• Web Interface with drag-and-drop file upload
• Docker Support with GPU acceleration

Quick Start

Option 1: Docker (Recommended)

# Clone the repository
git clone https://github.com/your-username/taltech-asr.git
cd taltech-asr

# Copy environment template and add your HuggingFace token
cp .env.example .env
# Edit .env and set HF_TOKEN (see Token Setup below)

# Start with Docker Compose
docker compose up taltech-asr

Visit http://localhost:8000 to access the web interface.

Option 2: Local Development

Prerequisites

• Python 3.12+
• uv package manager
• FFmpeg
• CUDA 12.1+ (optional, for GPU acceleration)

Installation

# Clone and setup
git clone https://github.com/your-username/taltech-asr.git
cd taltech-asr

# Install dependencies
uv sync

# Configure environment
cp .env.example .env
# Edit .env and set HF_TOKEN (see Token Setup below)

# Start the server
uv run uvicorn app.main:app --host 0.0.0.0 --port 8000

HuggingFace Token Setup

Speaker diarization requires a HuggingFace token with access to gated models:

1. Create a token at https://huggingface.co/settings/tokens
2. Accept the user conditions for these models:
   • https://huggingface.co/pyannote/speaker-diarization-3.1
   • https://huggingface.co/pyannote/segmentation-3.0
3. Add the token to your .env file:

   HF_TOKEN=hf_your_token_here
   

Usage

Web Interface

1. Visit http://localhost:8000
2. Drag & drop or select your audio/video file
3. Wait for processing (progress updates automatically)
4. Download results in your preferred format

Supported formats: MP3, WAV, MP4, AVI, MOV, M4A, AAC, OGG, FLAC, and more (up to 500MB)

API Usage

# Upload file for transcription
curl -X POST -F "file=@audio.mp3" http://localhost:8000/asr/transcribe/upload

# Check job status
curl http://localhost:8000/asr/job/{job_id}/status

# Download results
curl http://localhost:8000/asr/job/{job_id}/result?format=json -o result.json
curl http://localhost:8000/asr/job/{job_id}/result?format=srt -o result.srt

API Endpoints

Endpoint	Method	Description
/	GET	Web upload interface
/asr/transcribe/upload	POST	Upload file for transcription
/asr/transcribe/url	POST	Transcribe from URL
/asr/job/{job_id}/status	GET	Check job status
/asr/job/{job_id}/result	GET	Download results
/health	GET	Health check
/docs	GET	Interactive API documentation

Configuration

Key settings in .env:

# ASR Settings
ASR_OUTPUT_DIR=/tmp
ASR_TEMP_DIR=/tmp/taltech-asr
WHISPER_DEVICE=auto          # auto, cpu, cuda

# HuggingFace Token (required for diarization)
HF_TOKEN=hf_your_token_here

# Speaker Diarization
DIARIZATION_ENABLED=true
DIARIZATION_MAX_SPEAKERS=3   # Optional: limit max speakers

# Speech Separation (optional, disabled by default)
OVERLAP_HANDLING_ENABLED=false  # Set to true to enable MossFormer2 separation

Speech Separation (Optional)

By default, overlapping speech regions use mixed audio. To enable speech separation:

OVERLAP_HANDLING_ENABLED=true

This uses MossFormer2 to separate overlapping speakers, which improves transcription accuracy for conversations with frequent interruptions. Note: requires additional GPU memory.

Docker Deployment

# Development with live reload
docker compose up taltech-asr

# Production
docker compose --profile production up taltech-asr-prod

# Manual run with GPU
docker run -d --name taltech-asr \
  --gpus all \
  --shm-size=16g \
  --env-file .env \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/home/app/.cache/huggingface \
  taltech-asr:latest

Required Docker settings:

• --shm-size=16g - Required for Ray object store
• --env-file .env - Loads HF_TOKEN and configuration
• HuggingFace cache mount - For model access

Project Structure

taltech-asr/
├── app/
│   ├── main.py              # FastAPI application
│   ├── config.py            # Configuration
│   ├── routers/             # API endpoints
│   ├── workflows/           # Processing pipeline
│   ├── tasks/               # Ray workers
│   ├── workers/             # Actor management
│   └── templates/           # Web interface
├── models/                  # Downloaded ML models
├── Dockerfile
├── docker-compose.yml
└── pyproject.toml

Models

Models are downloaded automatically on first run:

Model	Purpose	Size
TalTechNLP/whisper-large-v3-turbo-et-verbatim	Estonian ASR	~1.6GB
openai/whisper-large-v3-turbo	Generic ASR	~1.5GB
pyannote/speaker-diarization-3.1	Speaker diarization	~200MB
speechbrain/lang-id-commonlanguage_ecapa	Language detection	~100MB

Slurm Deployment

For HPC cluster deployment, see SLURM_DEPLOYMENT.md.

Troubleshooting

Model download fails:

curl -X POST http://localhost:8000/models/download/all

GPU not detected:

# Check CUDA availability
python -c "import torch; print(torch.cuda.is_available())"
# Use CPU mode
WHISPER_DEVICE=cpu uv run uvicorn app.main:app

Docker Ray errors: Ensure --shm-size=16g is set for sufficient shared memory.

License

MIT License