Skip to content

fix(cli): turn off onnxruntime telemetry for Kokoro synthesis - #5425

Merged
jrusso1020 merged 1 commit into
heygen-com:mainfrom
thegreatLUCY:fix/tts-disable-onnxruntime-telemetry
Oct 13, 2026
Merged

jrusso1020 merged 1 commit into
heygen-com:mainfrom
thegreatLUCY:fix/tts-disable-onnxruntime-telemetry

Conversation

@thegreatLUCY

Copy link
Copy Markdown
Contributor

What

Run the Kokoro synthesis subprocess with ORT_DISABLE_TELEMETRY=1.

Why

onnxruntime's 1DS telemetry is on by default in its macOS Python wheels. When an upload response arrives while the Python process is exiting, the 1DS HTTP worker locks a static mutex that exit already destroyed and the process aborts, after the WAV was written. execFileSync throws, hyperframes tts exits 1, and the audio skill drops the line as a non-fatal anomaly, so narration can silently lose a sentence. Details, crash stacks and evidence in #5424.

Related work

Fixes #5424. Related: #2775 (narration dropped through a different cause).

How

One line: env: { ...process.env, ORT_DISABLE_TELEMETRY: "1" } on the execFileSync call in packages/cli/src/tts/synthesize.ts. onnxruntime reads the variable once when it loads, so the 1DS SDK never starts in the synth process; the rest of the environment is passed through unchanged (HYPERFRAMES_PYTHON, PATH, etc. are unaffected). The explanation lives here rather than in a code comment, per the comment-share ratchet.

Test plan

  • Unit tests added/updated: synthesize.cache.test.ts asserts the synth call passes ORT_DISABLE_TELEMETRY: "1". It fails on main ("expected undefined to be '1'") and passes with the change; all 31 tests in packages/cli/src/tts/ pass (vitest 4.1.11).
  • Manual testing performed: sampled the synth process 18 s into a run (macOS 15.7.7, onnxruntime 1.30.0, kokoro-onnx 0.4.7): 10 threads including the 1DS HTTP worker by default; 4 threads and no 1DS code with the variable set. Synthesis output is unchanged.
  • Documentation updated (if applicable): not applicable.
  • Comments follow CONTRIBUTING.md "Comments": no comments added; check-comment-citations.mjs OK and comment-ratchet.mjs at or under the base share.

🤖 Generated with Claude Code

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at a4e8b4e7. Thanks for the clear write-up in #5424.

What I checked

  • The switch is real in onnxruntime 1.30.0. At the v1.30.0 tag, core/platform/posix/telemetry.cc checks IsTelemetryDisabledByEnvironment() when the 1DS provider starts up. With ORT_DISABLE_TELEMETRY set to a truthy value, it returns before the uploader is created, and the decision holds for the life of the process. docs/Privacy.md documents the same thing: set it "before ONNX Runtime initializes". Setting it in the child's environment satisfies that, because it's in place before Python imports onnxruntime. ort.disable_telemetry_events() wouldn't be enough here, since it only suppresses non-essential events after the SDK is already running.
  • One place covers every Kokoro path. media-use/lib/tts-local-provider.mjs and skills/media-use/audio/scripts/lib/tts.mjs both shell out to hyperframes tts, so they both reach this execFileSync.
  • The rest of the environment passes through. { ...process.env, ... } keeps HYPERFRAMES_PYTHON, PATH and the rest.
  • The test fails without the fix. All 31 tests in src/tts/ pass at this head. With the env line removed, the new test fails with "expected undefined to be '1'".

Nit (not blocking): hasPythonPackage(python, "kokoro_onnx") runs import kokoro_onnx without the variable, and that loads onnxruntime too. I haven't checked whether importing alone starts the uploader. That's a short-lived process, though, so the exit race is much less likely there, and if it did crash the user would get a loud "not installed" error rather than a dropped line. Passing the same env there would rule it out.

No conflicts with main. When I posted this, CI showed 20 passed and 16 still running.

I didn't run this on macOS or reproduce the SIGABRT.

— Rames

@jrusso1020

Copy link
Copy Markdown
Collaborator

Thanks for this! It's approved and CI is green, but main only accepts signed commits, and your commit a4e8b4e7 isn't signed, so it can't merge yet.

Quickest fix with an SSH key you already use for GitHub:

git config --global gpg.format ssh
git config --global user.signingkey ~/.ssh/id_ed25519.pub   # your public key
git config --global commit.gpgsign true
git commit --amend --no-edit -S
git push --force-with-lease

Also add that same key to GitHub as a Signing key (Settings → SSH and GPG keys → New SSH key → Key type: Signing key), and make sure the commit's email is a verified email on your account. Docs: Signing commits and Telling Git about your signing key.

Once the commit shows Verified, we'll re-approve and merge.

— Rames

onnxruntime's 1DS telemetry, on by default in its macOS wheels, can abort the
Python process at exit when an upload response races teardown. execFileSync
then throws for a line whose WAV was already written, and the audio skill drops
it as non-fatal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
auto-merge was automatically disabled October 13, 2026 02:43

Head branch was pushed to by a user without write access

@thegreatLUCY
thegreatLUCY force-pushed the fix/tts-disable-onnxruntime-telemetry branch from a4e8b4e to dd0c18a Compare October 13, 2026 02:43
@thegreatLUCY

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review, Rames! The commit is now signed and shows as Verified. I'll leave the hasPythonPackage nit for a possible follow-up so this one stays as reviewed.

@jrusso1020
jrusso1020 enabled auto-merge October 13, 2026 04:02
@jrusso1020
jrusso1020 added this pull request to the merge queue Oct 13, 2026
Merged via the queue into heygen-com:main with commit f25cfbd Oct 13, 2026
81 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tts (Kokoro, macOS): onnxruntime's telemetry aborts Python at exit, so synthesized lines are reported as failed and dropped from the narration

2 participants