Skip to content

About

OpenClaw / Home Assistant voice assistant for the Voice PE — realtime conversation, custom wake words, speaker recognition, voice memory, and long-running task delegation that reports back by voice

Resources

Contributing

Stars

102 stars

Watchers

2 watching

Forks

Repository files navigation

Voice PE Realtime

Turn a Home Assistant Voice PE into the voice assistant you actually wanted — natural speech-to-speech conversation powered by OpenAI Realtime or GPT-Live, smart-home control with per-turn latency you can see in Home Assistant, a wake word you can retrain on your household's voices, an assistant that knows who's speaking, remembers what you tell it, and finds you when long-running work is done. Built on Home Assistant / OpenClaw: Home Assistant runs your home, OpenClaw is its memory, its hands, and its phone.

It runs on your Home Assistant box. Wake-word detection runs on the device; speaker identification, voice prints and any recordings stay on your box. OpenAI hears you only after a wake, and receives what the conversation needs: your instructions, your memory notes and tool results — exactly what leaves your network.

New in 0.19.0: GPT-Live, as an opt-in voice runtime

The add-on can now run on OpenAI GPT-Live (gpt-live-1) instead of the Realtime API. It is experimental and off by default: voice_runtime is unset on every existing install, nothing changes until you set it, and the Realtime path is untouched by the switch. Under voice_runtime: live:

  • Router mode. The add-on, not the voice model, decides and runs every tool. Home Assistant's own intent engine is tried first for plain device commands (locally, on safe domains only); anything else goes to a Responses backend called with the full tool set. The voice keeps talking while that happens, and speaks only results a tool verified — a success claim with no tool behind it is never played.
  • It knows the home. Home Assistant's exposed-entity overview, floors, areas and aliases sit in the backend's cached prompt prefix, refreshed every five minutes and at each wake, so loose wording ("the upstairs bedroom") lands on real names.
  • It knows the room. "Turn on the lights" acts on the room of the Voice PE that heard it, found through the device registry. Optionally (live_auto_expose, off by default) new controllable entities are exposed to Assist for you; helpers and scripts only when they are on a dashboard, locks never.
  • Long tasks finish in the background. A request that needs a slow tool (an OpenClaw agent turn, a web search) becomes a background task after a few seconds; the conversation goes on and the answer is delivered when it lands — in the session, as an announcement, or by text.
  • Cost controls, because GPT-Live bills by the clock. GPT-Live costs $0.05 per connected session minute, billed per second, idle time included. The add-on therefore opens a session only on a wake and closes it within seconds of quiet, and refuses new sessions past a hard daily cap (60 session-minutes / $3.00 by default, shared by all devices, persisted across restarts). Backend tokens are billed separately by OpenAI.
  • Unchanged: locks, garages, gates, doors and alarms still need a spoken yes, enforced below the model; a state question never runs a control tool.

To try it on one instance, add this in the add-on's ⋮ → Edit in YAML view and restart:

voice_runtime: live
live_acknowledge_gaps: true

Read GPT-Live cost safety first, then the add-on documentation's GPT-Live section, the Live options and the release notes. Remove the two lines and restart to go back. Realtime (gpt-realtime-2) stays the default and the recommended runtime; it is billed per response and costs nothing while idle.

What it feels like

"What's Grandma's number?" Recall from your household's long-term memory, without waiting for a full agent turn. Facts you teach it by voice are stored on your machine and become part of its instructions; connect OpenClaw and it reaches everything your agent remembers too.

"Research flight prices to London for October." "I'll look into it and report back." It hands the task to OpenClaw, which browses the web in the background for as long as it takes — then the result is announced out loud in the room you asked from, or texted to you if you've stepped out. Long-running tasks that find you when they're done.

Walking into the kitchen: "Set a pasta timer for 9 minutes." Nine minutes later: "Alex, your pasta timer is done" — spoken personally to whoever set it. A gentle bell follows only if nobody responds. Dismiss with a word or the button.

It knows who's speaking. On-device wake word, local voice recognition. It can greet you by name, keep per-person context, and restrict chosen tools to specific speakers — enforced below the model, so it can't be talked around. Voice matching is a convenience, not a lock: a similar voice can pass.

"Remember that we park at the north lot." Teach it standing rules by voice. They persist, attributed to whoever said them, and changes need a recognized household voice. "Forget that" removes them; "what do you remember?" reads them back.

And it just converses. Speech in, speech out — no STT→LLM→TTS chain, so tone and timing feel human. Interrupt it mid-sentence with "stop". Follow up without repeating the wake word.

"Unlock the front door." "Do you want me to unlock the front door?" Locks, garage doors, gates and alarm panels need a spoken yes — enforced in the add-on, not left to the model's judgement.

And you can see how fast it is. Every turn's timeline — wake to mic open, end of speech to first reply audio, each tool's duration — is published to Home Assistant, so you can measure your own setup instead of trusting a number in a README.

What people do with it

Marked † = needs the optional agent integration — built for OpenClaw, works with any agent. Everything else is built in.

  • "What's the wifi password?" — say "remember the wifi password is…" once, and it's answered from then on. Same for the pool gate code, shoe sizes, where the spare key lives.
  • "When's Grandma's birthday?" † — recall from OpenClaw's long-term memory (a text search, no agent turn).
  • "What did we decide about the fence contractor?" † — decisions and history, not just facts.
  • "Text Sam we're running ten minutes late." † — hands covered in flour; OpenClaw sends it through any of its channels (iMessage, Telegram, WhatsApp, …).
  • "Call the pharmacy and ask if my prescription is ready, then tell me what they say." † — pair it with OpenClaw and my openclaw-voice-call-realtime plugin, which gives your assistant a real phone: it places the call, runs the errand, and the answer is spoken back in the room you asked from.
  • "Research flights to Tokyo in October and text me the three best options." † — acknowledged now, browsed in the background for as long as it takes, delivered when done.
  • "Add everything for lasagna to the shopping list." — native Home Assistant list tools. Then "set a pasta timer" — dismissed or delivered by name when it's done.
  • A voice for your automations. † — the announce endpoint accepts any authorized POST, so OpenClaw's scheduled jobs (or any script on your LAN) can speak in the room: "leave in fifteen minutes for the school run."

Longer versions, with the how-it-works behind each: Stories.

Features

  • OpenAI speech-to-speech — gpt-realtime-2 by default; optional GPT-Live (gpt-live-1) keeps talking while a separate Responses backend reasons and runs tools. The Live runtime is opt-in while it completes its release soak. Note that GPT-Live is billed per connected session minute ($0.05/min, billed per second) and idle time counts, so Live sessions here are opened by a wake and closed again within seconds, under hard daily spend caps. Realtime is billed per response and costs nothing while idle.
  • Native Home Assistant control via the official MCP Server integration — scoped to exactly the entities you expose
  • Custom wake word — "Hey Leonard" ships as the default (trained by this project); switch to Hey Jarvis / Okay Nabu from a dropdown in HA, or train your own
  • Speaker recognition — local voice-print identification with guided voice enrollment (say "train my voice")
  • Voice-instructed memory — "remember…" / "forget…" / "what do you remember?", speaker-gated writes
  • OpenClaw integration — recall_memory searches your agent's memory directly, deep questions escalated to a full agent turn; a ready-to-run bridge ships in this repo (contracts are agent-agnostic)
  • Long-running task delegation — OpenClaw reports back by voice, in the room that asked, via the announce endpoint
  • Voice timers — personal announcement → grace period → gentle bell, dismissed by button or voice
  • False-wake flagging — by voice or double-press, tied to the exact device and wake; feeds a gated wake-word learning loop that never ships a model worse than the one you have
  • Web search — current info via a single extra OpenAI call (on by default)
  • HA sensors — per-turn latency (with p50/p90), the active wake-word model and threshold, current speaker, active timers, wakes and false wakes today, enrollment active
  • Persona fully yours — rewrite the instructions and choose from OpenAI's built-in voices
  • Safety — device token for the WebSocket, spoken-yes confirmations for locks/garage/gates/alarm, privacy controls for what is stored (nothing beyond counters by default)
  • Production hardening — proactive session refresh before OpenAI's 60-minute cap, reconnect recovery that replays the request you were making, spoken error messages, echo/ghost-turn guards, stop-word authority, turn-liveness watchdogs

Architecture at a glance

Home Assistant Voice PE           Home Assistant (your box)              Cloud
┌─────────────────────────┐   WS   ┌──────────────────────────┐   WS   ┌──────────────┐
│ custom ESPHome firmware │ ─────▶ │ this add-on              │ ─────▶ │ OpenAI       │
│ wake word + XMOS DSP    │ 16 kHz │ (session, tools, memory, │ 24 kHz │ Realtime or  │
│ thin audio client       │ ◀───── │  speaker ID, timers)     │ ◀───── │ GPT-Live     │
└─────────────────────────┘        └───────────┬──────────────┘        └──────────────┘
                                               │ tools
                                               ▼
                              HA MCP Server → controls your home
                              OpenClaw    ←→  recall / delegate / announce (optional)

Three parts:

  1. Firmware (voicepe-realtime-firmware) — turns the Voice PE into a thin, low-latency audio client. Wake word runs on-device.
  2. Backend add-on (this repo) — owns the OpenAI Realtime or GPT-Live session, Home Assistant tools, speaker identity, timers, and memory.
  3. OpenClaw integration (optional) — deep recall, messaging, calls, and long-running task delegation (agent-agnostic contracts). Everything else works without it.

Quick start

  1. Install the add-on: Settings → Add-ons → Add-on Store → ⋮ → Repositories → add https://github.com/TristanBrotherton/voicepe-realtime → install OpenAI Realtime 2 Voice Agent. Set your OpenAI API key.
  2. Give it your home: add Home Assistant's MCP Server integration and expose the entities you want voice-controlled to Assist.
  3. Flash the firmware: adopt your Voice PE in ESPHome Builder and paste in the device stub from the firmware repo. First flash over USB, updates OTA.
  4. Say "Hey Leonard" and ask for a light.

Full walkthrough (~30–45 minutes from zero): Getting Started.

Documentation

Guide What's in it
Getting Started Prerequisites, flashing, add-on install, first conversation, multi-device
Stories What households actually do with it — and which feature makes each one work
Configuration Reference Every add-on option and firmware substitution — purpose, default, when to change it
Features Wake words, speaker recognition, memory, timers, false-wake flagging, latency, confirmations, web search, sensors, persona
Wake-word learning How false wakes become better models without shipping regressions: capture modes, labels, the release gate, shadow and canary
Demo A scripted, reproducible demo with timing — runs against a simulated device, no household needed
Agent Integration The bridge contracts: recall, escalation, and the announce endpoint — works with any agent
FAQ Cost, privacy (what leaves your network), reverting to stock, Raspberry Pi, languages, and more
Release notes What changed in 0.19.1 (explain mode) and 0.19.0 (GPT-Live), and what to check before enabling it; the full history is in the changelog
Contributing PRs welcome — small, tested, explained

The firmware lives in its own repo: TristanBrotherton/voicepe-realtime-firmware. Firmware 1.3.1 (its current main) works with both runtimes; GPT-Live needs no firmware change.

Credits

License

MIT. The firmware repo carries the upstream ESPHome license.

About

OpenClaw / Home Assistant voice assistant for the Voice PE — realtime conversation, custom wake words, speaker recognition, voice memory, and long-running task delegation that reports back by voice

Resources

Contributing

Stars

102 stars

Watchers

2 watching

Forks

Releases

Packages

Used by

Contributors

Languages