daemon: inbound-path watchdog — auto-recover the long-uptime NAT wedge#369
Merged
Conversation
…wedge A daemon could run for days transmitting into a stale NAT/relay mapping while receiving nothing (2026-07-13 incident: 43.5 MB sent vs 102 KB received over 2d19h, every send-message failing with "cannot connect (data exchange port 1001)"). The registry heartbeat is TCP and kept succeeding, so trustRepublishLoop never noticed and only a manual restart cleared it. The new watchdog (pkg/daemon/rxwatchdog.go) samples PktsRecv/PktsSent every 30s. On 3 min of delivered-packet silence with active transmit it soft-recovers: RegisterWithBeacon (the discover reply doubles as an active inbound probe) plus registry reRegister. If the wedge survives 3 soft attempts it exits with code 86 so launchd (KeepAlive SuccessfulExit=false) / systemd (Restart=always) respawn the daemon with a fresh transport — the remedy that demonstrably clears the wedge. Flap guards on the hard exit: never fires when the registry is also unreachable (machine offline — restart fixes nothing), when inbound never progressed this process (would boot-loop), or within 30 min of start. Emits tunnel.rx_silence / tunnel.rx_recovered / tunnel.rx_wedged_exit. Disable with -no-rx-watchdog. Supporting changes: - tunnel.go: LastRecvNano stamped on every raw datagram + LastRecvTime() accessor, so logs separate dead-socket from dead-session-layer wedges. - daemon.go: lastRegistryOKNano stamped on registration/heartbeat/ re-registration successes; startTime now set at the top of Start() so watchdog goroutine reads never race the historical late assignment. - pilotctl daemon start: launchd readiness wait 10s -> 30s — app-store apps spawn before IPC comes up, and the old timeout reported a false failure for boots that succeed moments later. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
govulncheck flags GO-2026-5856 (crypto/tls Encrypted Client Hello privacy leak), fixed in the standard library at go1.25.12. CI installs the toolchain via go-version-file: go.mod, so bumping the go directive pulls in the fixed stdlib and clears the vuln repo-wide. Build verified under go1.25.12. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ypto) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A daemon can run for days transmitting into a stale NAT/relay mapping while receiving nothing. 2026-07-13 incident on cascade-test: 43.5 MB / 572K pkts sent vs 102 KB / 1,443 pkts received over 2d19h, every
send-messagefailing withcannot connect (data exchange port 1001). The registry heartbeat is TCP and kept succeeding, sotrustRepublishLoopnever noticed — only a manual restart cleared it.Fix
New inbound-path watchdog (
pkg/daemon/rxwatchdog.go), on by default:PktsRecv/PktsSentevery 30s; wedge = 3 min of delivered-packet silence while ≥30 packets went out. NewLastRecvNanoraw-datagram timestamp distinguishes dead-socket from dead-session-layer in logs.RegisterWithBeacon(the discover reply doubles as an active inbound probe) + registryreRegister.KeepAlive.SuccessfulExit=false) / systemd (Restart=always) respawn with a fresh transport — the remedy that demonstrably clears the wedge.tunnel.rx_silence/tunnel.rx_recovered/tunnel.rx_wedged_exit. Opt out:-no-rx-watchdog.Also:
pilotctl daemon startlaunchd readiness wait 10s→30s (app-store apps spawn before IPC; the old timeout reported a false failure for boots that succeed moments later).Testing
-race -count=5.pkg/daemonsuite: only the two pre-existing failures (TestWriteLoopExitsOnWriteDeadline,TestIPCServer_MaxClientsCap— confirmed pre-existing viagit stash).🤖 Generated with Claude Code