Orchestrator's Temporal worker never retries a failed initial connection — silently stops polling forever, no crash, no log after the first attempt

[ptigroup]

Orchestrator's Temporal worker never retries a failed initial connection — silently stops polling forever, no crash, no log after the first attempt

Versions: nestjs-temporal-core@3.2.3, @temporalio/worker@1.15.0 (self-hosted, docker-compose deployment)

What happened

After a docker compose restart that raced Temporal's own startup, the orchestrator app's Temporal worker failed its one connection attempt at boot:

[TemporalWorkerManagerService] ERROR Failed to create connection
TransportError: tcp connect error, <temporal-ip>:7233, Connection refused

The NestJS app then logged Nest application successfully started and kept running normally — HTTP health endpoint up, pm2/process manager showing it "online" with 0 restarts — but the Temporal worker never registered a poller on any task queue again. Scheduled posts (workflows already queued) sat untouched indefinitely, because nothing was ever polling for them. This condition lasted 31 hours before being noticed, and is a repeat of an earlier ~week-long occurrence of the same failure mode.

Root cause

initializeMultipleWorkers() (and the single-worker initializeWorker() path) in TemporalWorkerManagerService treats a connection failure at boot as tolerable by default (allowConnectionFailure !== false): it logs a warning/error and returns, rather than retrying or throwing. onModuleInit() runs exactly once, at process startup — there is no later code path (no interval, no scheduled retry, no reconnect-on-next-use) that ever attempts the connection again. A transient "Temporal isn't ready yet" condition at boot therefore becomes a permanent loss of worker functionality for the life of the process, indistinguishable from healthy at the process-supervisor level.

Why this is worse than a normal transient-connection issue

Most supervisors (pm2, Docker, k8s) recover automatically from a crashed process. This failure mode specifically avoids that safety net: the process doesn't crash, so nothing restarts it, and its own health endpoint doesn't reflect the missing worker either (in Postiz's case, /health/status opens an independent fresh Temporal connection to test Temporal's own reachability — not whether the worker's registered connection is actually polling — so it reports ok throughout an outage exactly like this one).

Suggested fix

Either (or both):

1. Retry the initial connection with backoff inside onModuleInit()/initializeMultipleWorkers() instead of giving up after one attempt.
2. Provide (or default to) a mode where a failed initial connection is treated as fatal (throws instead of silently returning), so the host process supervisor's own crash-restart behavior provides the retry loop for free.

Happy to provide the exact getTemporalModule() call site in Postiz's own code if useful — it never sets allowConnectionFailure, so it inherits this library's silently-tolerant default for worker registration specifically (not for the simpler client-only connections elsewhere in the app, which don't go through this same code path and were unaffected during this incident).

[JoseAlmeida2009]

We hit this exact symptom on a self-hosted instance — posts sitting in QUEUE past their time, no errors anywhere, pm2 reporting the orchestrator as online with 0 restarts — and traced it to a second root cause that produces an identical failure. It may explain the cases where the timing rules out a startup race, or where a restart does not help.

In our case it was not the Temporal connection: the process never got far enough to attempt one.

main.ts calls initializeSentry() as its first statement, and initialize.sentry.ts imports @sentry/nestjs and @sentry/profiling-node at the top level — so both load on every start, whether or not NEXT_PUBLIC_SENTRY_DSN is set (the function itself returns null immediately when it isn't). Loaded in that order they deadlock the module loader: the instrumentation hook @sentry/nestjs installs on require races the dlopen that @sentry/profiling-node performs for its native addon.

Measured on our instance, using ephemeral node -e processes inside the running container so the publishing queue was never touched:

what was loaded	blocked
@sentry/profiling-node alone	0 / 25
@sentry/nestjs alone	0 / 60
both, via initialize.sentry.js	2 / 105 (~1.9%)
both, with the requires moved below the DSN guard	0 / 105

Neither module blocks on its own. It is the combination.

How to tell this apart from a failed Temporal connection

A stuck process looks like this:

wchan:   futex_wait_queue     ← all 11 threads, the 4 libuv workers included
Threads: 11                   ← a healthy worker has ~135; 11 is an empty Node runtime
utime/stime identical across two reads 5s apart   ← 0% CPU, so not slow startup

Counting threads is the cheapest check there is, and pm2 list will not tell you: the process is online either way. The giveaway in the logs is that Starting Nest application is never written at all — the process stops before Nest boots, which is why there is no Temporal error to find. From Temporal's side, a worker that never started is indistinguishable from one that started and stopped, which is what sent us looking in the wrong place for a month.

Ruled out with evidence along the way: OOM (OOMKilled=false, no oom-kill in dmesg, 56 GB free), slow startup (0% CPU over five minutes is a block, not slowness), and the Temporal addon itself (@temporalio/core-bridge loaded 30/30 without blocking).

Fix

Move both requires below the if (!process.env.NEXT_PUBLIC_SENTRY_DSN) return null guard. Running in production here since today — the first three scheduled posts after the restart published on time, and the queue has no stragglers.

Worth stating plainly: as it stands, every deployment pays a ~2% chance of a silent startup deadlock for an integration that most self-hosters never configure. On a scheduler, the cost of that is not an error page — it is a calendar that looks full while nothing goes out. Ours lost 97 posts over nine days before anyone noticed.

Happy to open a PR if that's useful. Note the current main has since grown setSentryUserContext, which also uses Sentry, so the fix needs a shared lazy loader rather than moving the imports into one function.

[ptigroup]

Thanks for digging into this — really solid isolation work (the ephemeral-process block-rate testing and the wchan/thread-count check are both great, cheap diagnostics).

Can confirm this is a real, distinct failure from what we filed the issue for: our incident had a logged Temporal ECONNREFUSED before going quiet, meaning Nest had booted and reached onModuleInit(). Your case never gets there at all — no Starting Nest application line — so these are two different mechanisms producing the same outward symptom (orchestrator "online" per pm2, zero restarts, queue not draining, no error anywhere).

Worth noting for anyone else landing here: we checked, and NEXT_PUBLIC_SENTRY_DSN doesn't need to be set for this to bite — the guard only gates the function's return value, the imports run regardless. So this affects every self-hosted deployment, not just ones with Sentry configured, which matches what you said about the ~2% tax being paid by people who never turned Sentry on.

For anyone hitting this before a real fix lands: an external restart (pm2 restart orchestrator, or a full docker restart temporal postiz if that's not enough) recovers from either root cause — ours does this automatically now, no manual intervention needed once detected. Your thread-count check is the right way to tell which of the two you actually hit, for anyone who wants to know rather than just recover.

+1 to opening a PR — a shared lazy loader (given setSentryUserContext now also touches Sentry elsewhere) sounds like the right shape.

[p-oancia]

Confirming this on a self-hosted deployment, with one detail that matters for the workaround suggested in #1748.

Environment: Postiz self-hosted, docker-compose, 8 containers (app, postgres, redis, Temporal + its own postgres/elasticsearch), Ubuntu 24.04, Docker 29.8.1.

Occurrence 1 (unnoticed for 38 hours). The Docker daemon itself restarted (unattended package upgrade). Everything came back, and the app logged:

[TemporalConnectionFactory] Successfully connected to temporal:7233
[TemporalConnectionFactory] WARN Failed to create worker connection, returning null
   TransportError: tonic::transport::Error(Transport, ConnectError(ConnectError("tcp connect error", <temporal-ip>:7233, Os { code: 111, kind: ConnectionRefused })))
[TemporalWorkerManagerService] ERROR Failed to create connection

Note the client connection succeeds and the worker connection, a fraction of a second later, does not. After that the process ran for 38 hours with no workers: container healthy, HTTPS 200, pm2 showing orchestrator online with 0 restarts, nothing further in the logs. Posts sat in QUEUE with no error written. The only reliable signal was temporal task-queue describe --task-queue main showing an empty Pollers list, with a backlog age matching the daemon restart exactly.

Occurrence 2 (deliberate, reproducible). Before rebooting the host I added, per the analysis in #1748:

depends_on:
  postiz-redis: { condition: service_healthy }
  postiz-postgres: { condition: service_healthy }
  temporal: { condition: service_healthy }

The reboot reproduced the fault anyway, with the identical trace. All containers came back healthy and the app again had zero pollers. So gating on Temporal's container health is not sufficient: Temporal reports healthy before its gRPC frontend accepts worker connections, and the app's single attempt still lands inside that window. Worth knowing for anyone who adopts the depends_on workaround and assumes they are covered.

What would fix it: retry the worker connection in TemporalWorkerManagerService with backoff instead of one attempt, or have the orchestrator's /health/status reflect worker registration rather than opening a fresh client connection each call. As it stands, the health endpoint reports ok precisely when the process is unable to do its job, which is what makes this silent.

Workaround, if useful to others. Since the app cannot recover on its own, we run a cron watchdog every 10 minutes that restarts the container when the main task queue has no pollers or a post is more than 10 minutes past its publish date, then verifies the workers registered:

pollers=$(docker exec temporal-admin-tools temporal task-queue describe \
  --address temporal:7233 --task-queue main | sed -n '/Pollers/,$p' | grep -c UNVERSIONED)
[ "$pollers" -eq 0 ] && (cd /opt/postiz && docker compose restart postiz)

It fired on the reboot above: fault at 00:20:01, workers polling again at 00:20:58. Queued posts publish within about a minute of the workers returning. Happy to share the full script if it would help anyone.