Connection lifecycle logs, long-wedge escalation, context on remaining warns - #18
Open
petergaultney wants to merge 1 commit into
Open
Conversation
petergaultney
added a commit
to petergaultney/relay-server
that referenced
this pull request
Aug 3, 2026
…remaining warns Port of the same changes on thds/connection-lifecycle-logs (upstream PR No-Instructions#18) to the v0.12.1 base: - info-level 'WebSocket disconnected' with doc_id, user, close_reason, and duration (connect logged at debug). - while an outbound channel stays full, re-warn every 5 minutes with the cumulative drop count and how long it has been full; the recovery info line reports the full duration. - doc_id/user on 'Error handling message' and 'Received non-binary message'; user on the subscription-lock and Subdocs/Event protocol warns. - the read loop now breaks on stream EOF (as upstream main does) instead of waiting on server shutdown with the branch disabled; without this the disconnect log would never fire for connections that die without a close handshake.
…remaining warns - info-level 'WebSocket disconnected' with doc_id, user, close_reason, and duration (connect logged at debug to keep volume down); makes wedged and reconnect-looping clients visible directly in pod logs. - while an outbound channel stays full, re-warn every 5 minutes with the cumulative drop count and how long it has been full; the recovery info line now also reports the full duration. - add doc_id/user to 'Error handling message' and 'Received non-binary message', and user to the subscription-lock and Subdocs/Event protocol warns in doc_connection.rs.
petergaultney
force-pushed
the
thds/connection-lifecycle-logs
branch
from
August 5, 2026 15:09
0bee174 to
21cae26
Compare
petergaultney
marked this pull request as ready for review
August 5, 2026 15:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three observability changes for diagnosing client-wedge incidents in pod logs:
Connection lifecycle log.
handle_socket_innerpreviously logged nothing when a websocket ended - the close reason was computed but only fed to metrics. Now an info-levelWebSocket disconnectedreportsdoc_id,user,close_reason, andduration_secs, so wedged clients and reconnect loops are visible directly in the logs (a reconnect loop reads as rapid short-duration disconnects from one user). Connect is logged at debug to keep steady-state volume at one line per connection.Long-wedge escalation. The full/recovered transition warns emit one line at the moment a channel fills - a client wedged for hours is silent after minute zero. While the channel stays full, re-warn every 5 minutes with the cumulative drop count and how long it has been full; the recovery line now also reports the total full duration.
Context on the remaining contextless warns.
Error handling messageandReceived non-binary messagegetdoc_id/user(the former fires per bad message, so it was the same defect class as the old backpressure warn); the subscription-lock and Subdocs/Event protocol warns indoc_connection.rsgetuser.No behavior changes;
cargo testpasses. We are running the same changes in production.