fix(hub): re-dial SSH-pull agents when a connection silently dies - #2126
Open
TowyTowy wants to merge 1 commit into
Open
fix(hub): re-dial SSH-pull agents when a connection silently dies#2126TowyTowy wants to merge 1 commit into
TowyTowy wants to merge 1 commit into
Conversation
The per-system updater reused an SSH client across ticks and ran the data exchange with no deadline. If a connection went half-open (dead peer that never sends RST/FIN) or an agent accepted the session but never wrote a response, the read in fetchDataViaSSH blocked forever. Because StartUpdater calls update() synchronously on its ticker, a blocked read froze the whole per-system goroutine: the ticker's subsequent ticks were dropped, no error was returned so the system stayed "up", and the agent was never re-dialed until the hub process restarted. Bound each SSH data exchange with sshOperationTimeout via runWithTimeout: on timeout the connection is torn down (unwinding the blocked read) and a retryable error is returned, so the next tick re-dials. Also enable TCP keep-alive on dialed connections as a backstop for genuine network death. Fixes henrygd#2041 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📃 Description
On the SSH-pull connection model, a per-system SSH connection could go half-open (a dead peer that never sends RST/FIN) or an agent could accept the session but never write a response. The hub's data read (fetchDataViaSSH → runSSHOperation) had no deadline, so it blocked forever. Because StartUpdater runs update() synchronously on its ticker, the blocked read froze the whole per-system goroutine — the ticker's later ticks were dropped, no error was returned (so the system stayed "up"), and the agent was never re-dialed until the hub process was restarted.
This matches #2041: agents silently drift to "dead-but-up", the hub logs no disconnect/redial, restart temporarily fixes everything, and which agents stall reshuffles between cycles.
Fix
runWithTimeout). On timeout the connection is torn down (unwinding the blocked read) and a retryable error is returned, so the next collection tick re-dials. The bound (20s) is kept under the 60s collection interval so a stalled agent recovers within one cycle.🪵 Changelog
🔧 Fixed
Fixes #2041