日本語 | 中文 | Español | Français | हिन्दी | Italiano | Português (BR)
Deterministic vocal instrument engine — additive synthesis, voice presets, real-time WebSocket streaming, multi-user jam sessions, cockpit UI
A deterministic vocal instrument engine built in TypeScript. Renders singing voices from score data using additive synthesis, voice presets, and real-time WebSocket streaming. Play live via keyboard/MIDI, collaborate in multi-user jam sessions, or render scores to WAV.
Status: v1.0.3 source — not yet published to npm. Install from source until v1.0.4 publishes (see Install).
- Additive vocal synthesis — harmonic partials + spectral envelope + noise residual
- 15 voice presets — frozen analysis artifacts from Kokoro TTS voices + lab presets, each with multiple timbres
- Polyphonic rendering — configurable max polyphony with per-voice state management and voice stealing
- Live mode — play notes via keyboard or MIDI with real-time WebSocket audio streaming
- Jam sessions — multi-user collaborative sessions with host authority, participant attribution, and recording
- Score input — load a
VocalScoreinto a track for automatic playback synced to transport - Recording & export — capture live performances to an EventTape, export to WAV with full provenance
- Lyrics & phonemes — grapheme-to-phoneme pipeline with phoneme lane visualization
- Cockpit UI — browser-based SPA with piano roll editor, live keyboard, XY pad, render bank, and telemetry
- Deterministic — seeded RNG, reproducible output from the same inputs
┌─── Cockpit UI (browser SPA) ───┐
│ Piano Roll │ Live │ Renders │
└──────────────┴────────┴─────────┘
│ │
REST API WebSocket
│ /ws /ws/jam
┌──────────┴────────┴─────────────┐
│ Express Server │
│ Render API │ Jam Sessions │
└──────┬──────┴───────┬────────────┘
│ │
StreamingVocalSynthEngine │
LiveSynthEngine ─────────┘
│
┌────────────┼─────────────┐
VoicePreset DSP (FFT) Curves (ADSR,
(.f32 blobs) Pitch Det. vibrato, automation)
Key directories:
| Directory | Purpose |
|---|---|
src/engine/ |
Core synth — block renderer, streaming engine, ADSR/vibrato curves |
src/dsp/ |
Signal processing — FFT, pitch detection |
src/preset/ |
VoicePreset schema, loader, and resolver |
src/server/ |
Express + WebSocket API server, jam session manager |
src/types/ |
Shared types — scores, jam protocol, presets |
src/cli/ |
User-facing CLI tools (analyze, build-preset, compare, inspect, play-score, resynth, gen-vowel-wav, realtime-demo) |
scripts/ |
Build/test regression scripts (not shipped, not part of npm test) |
apps/cockpit/ |
Browser cockpit UI (Vite + vanilla TS) |
presets/ |
15 bundled voice presets with binary timbre data |
The package @mcptoolshop/vocal-synth-engine is not yet published to npm. Until v1.0.4 ships, install from source:
git clone https://github.com/mcp-tool-shop-org/vocal-synth-engine.git
cd vocal-synth-engine
npm ci
npm run buildTo pin a specific commit in a downstream project:
npm install github:mcp-tool-shop-org/vocal-synth-engine#<commit-sha>npm ci
npm run devThe dev server starts at http://localhost:4321. The cockpit UI is served from the same port.
The cockpit is a browser-based SPA with three tabs:
- Piano roll with drag-to-create, move, and resize notes (C2-C6 range)
- Per-note controls: velocity, timbre, breathiness, vibrato, portamento
- Lyrics input with automatic phoneme generation
- Phoneme lane overlay synced to the piano roll
- Render to WAV with configurable preset, polyphony, seed, and BPM
- 24-key chromatic keyboard (mouse + key bindings)
- MIDI device input with channel filtering
- XY pad for real-time timbre morphing (X) and breathiness (Y)
- Hold pedal, velocity/breathiness sliders, vibrato controls
- Metronome with quantize grid (1/4, 1/8, 1/16)
- Latency calibration (low/balanced/safe presets)
- Record performances and save to render bank
- Live telemetry: voices, peak dBFS, RTF, click risk, WS jitter
- Browse, play, pin, rename, and delete saved renders
- Load a render's score back into the editor
- Side-by-side telemetry comparison between renders
- Provenance tracking: commit SHA, score hash, WAV hash
Multi-user collaborative sessions over WebSocket (/ws/jam):
- Host authority — session creator controls transport, tracks, recording, and quantization
- Guest participation — guests can play notes on any track but cannot modify session state
- Track ownership — tracks belong to their creator; only owner or host can modify/remove
- Participant attribution — every note event in the EventTape records who played it
- Score input mode — load a
VocalScoreinto a track for automatic playback synced to transport - Recording — capture all participants' notes into an EventTape, export to WAV
- Metronome — shared metronome with configurable BPM and time signature
Clients connect to /ws/jam and exchange JSON messages:
Client: jam_hello → Server: jam_hello_ack (participantId)
Client: session_create → Server: session_created (snapshot)
Client: session_join → Server: session_joined (snapshot)
Client: track_note_on/off → Server: track_note_ack
Client: record_start/stop → Server: record_status
Client: record_export → Server: record_exported (renderId)
Client: track_set_score → Server: score_status
| Endpoint | Method | Auth | Description |
|---|---|---|---|
/api/health |
GET | No | Server health, version, uptime |
/api/presets |
GET | No | List voice presets with timbres and metadata |
/api/phonemize |
POST | Yes | Convert lyrics text to phoneme events |
/api/render |
POST | Yes | Render a score to WAV |
/api/renders |
GET | Yes | List all saved renders |
/api/renders/:id/audio.wav |
GET | Yes | Download render WAV |
/api/renders/:id/score |
GET | Yes | Original score JSON |
/api/renders/:id/meta |
GET | Yes | Render metadata |
/api/renders/:id/telemetry |
GET | Yes | Render telemetry (peak, RTF, clicks) |
/api/renders/:id/provenance |
GET | Yes | Provenance (commit, hashes, config) |
Auth is optional — enabled when AUTH_TOKEN is set in the environment. Tokens can be supplied via Authorization: Bearer <token> header or ?token=<token> query parameter.
| Path | Purpose |
|---|---|
/ws |
Live mode — single-user note playback with audio streaming |
/ws/jam |
Jam sessions — multi-user collaboration with recording |
vocal-synth-engine ships an MCP (Model Context Protocol) server so Claude agents and other MCP clients can call the engine directly — no HTTP scaffolding needed. Boot via the vocal-synth-engine-mcp bin entry (stdio transport).
Tools exposed:
| Tool | Purpose |
|---|---|
render_score |
Render a VocalScore through a preset → base64 WAV + telemetry |
phonemize_text |
Lyrics text → ARPAbet PhonemeEvents (note-aligned if notes provided) |
list_presets |
Enumerate available preset ids (same shape as GET /api/presets) |
validate_score |
Parse + validate VocalScore JSON without rendering |
inspect_preset |
Preset manifest + per-timbre harmonics/energy (same as vse-inspect --json) |
Wire it into a Claude Desktop / Code config:
{
"mcpServers": {
"vocal-synth-engine": {
"command": "npx",
"args": ["-y", "@mcptoolshop/vocal-synth-engine", "vocal-synth-engine-mcp"]
}
}
}15 bundled presets with multi-timbre support:
| Preset | Voice | Timbres |
|---|---|---|
default-voice |
Baseline female | Default timbre |
bright-lab |
Lab/experimental | Bright formant |
kokoro-af-* |
Aoede, Heart, Jessica, Sky | Multiple per voice |
kokoro-am-* |
Eric, Fenrir, Liam, Onyx | Multiple per voice |
kokoro-bf-* |
Alice, Emma, Isabella | Multiple per voice |
kokoro-bm-* |
George, Lewis | Multiple per voice |
Each preset includes binary .f32 assets (harmonic magnitudes, spectral envelope, noise floor) and a JSON manifest describing pitch range, resonance, and vibrato defaults.
npm run dev # Dev server with hot reload
npm run build # Build cockpit + server
npm start # Production server
npm run inspect # CLI preset inspectorThe primary test surface is vitest:
npm test # Run all unit + integration tests once
npm run test:watch # Watch mode
npm run test:coverage # Coverage reportAdditional regression scripts under scripts/ (require a running dev server
for the jam tests; the others are standalone):
npx tsx scripts/test-jam-session.ts # Jam session lifecycle
npx tsx scripts/test-jam-recording.ts # Recording & export
npx tsx scripts/test-jam-collaboration.ts # Collaboration & score input
npx tsx scripts/test-score-render.ts # Score rendering pipeline
npx tsx scripts/test-consonants.ts # Consonant phonemes
npx tsx scripts/test-g2p.ts # Grapheme-to-phoneme
npx tsx scripts/test-lyrics-golden.ts # Lyrics golden tests
npx tsx scripts/test-multi-timbre.ts # Multi-timbre rendering
npx tsx scripts/test-noise-tail.ts # Tail silence/noiseThese are slated to migrate to tests/integration/ under vitest in a future
wave so they pick up npm test coverage automatically.
| Aspect | Detail |
|---|---|
| Data touched | Audio synthesis (in-memory), WebSocket connections (localhost), WAV file output, score data, voice presets |
| Data NOT touched | No telemetry, no analytics, no cloud sync, no credentials stored |
| Permissions | Network: WebSocket server on localhost. Disk: WAV file output to user-specified paths |
| Network | Localhost WebSocket server only — no outbound connections |
| Telemetry | None collected or sent |
See SECURITY.md for vulnerability reporting.
| Category | Score |
|---|---|
| A. Security | 10 |
| B. Error Handling | 10 |
| C. Operator Docs | 10 |
| D. Shipping Hygiene | 10 |
| E. Identity (soft) | 10 |
| Overall | 50/50 |
Full audit: SHIP_GATE.md · SCORECARD.md
MIT. See LICENSE.