Skip to content

Connection lifecycle logs, long-wedge escalation, context on remaining warns - #18

Open
petergaultney wants to merge 1 commit into
No-Instructions:mainfrom
petergaultney:thds/connection-lifecycle-logs
Open

Connection lifecycle logs, long-wedge escalation, context on remaining warns#18
petergaultney wants to merge 1 commit into
No-Instructions:mainfrom
petergaultney:thds/connection-lifecycle-logs

Conversation

@petergaultney

@petergaultney petergaultney commented Aug 3, 2026

Copy link
Copy Markdown

Three observability changes for diagnosing client-wedge incidents in pod logs:

  1. Connection lifecycle log. handle_socket_inner previously logged nothing when a websocket ended - the close reason was computed but only fed to metrics. Now an info-level WebSocket disconnected reports doc_id, user, close_reason, and duration_secs, so wedged clients and reconnect loops are visible directly in the logs (a reconnect loop reads as rapid short-duration disconnects from one user). Connect is logged at debug to keep steady-state volume at one line per connection.

  2. Long-wedge escalation. The full/recovered transition warns emit one line at the moment a channel fills - a client wedged for hours is silent after minute zero. While the channel stays full, re-warn every 5 minutes with the cumulative drop count and how long it has been full; the recovery line now also reports the total full duration.

  3. Context on the remaining contextless warns. Error handling message and Received non-binary message get doc_id/user (the former fires per bad message, so it was the same defect class as the old backpressure warn); the subscription-lock and Subdocs/Event protocol warns in doc_connection.rs get user.

No behavior changes; cargo test passes. We are running the same changes in production.

petergaultney added a commit to petergaultney/relay-server that referenced this pull request Aug 3, 2026
…remaining warns

Port of the same changes on thds/connection-lifecycle-logs (upstream PR No-Instructions#18)
to the v0.12.1 base:
- info-level 'WebSocket disconnected' with doc_id, user, close_reason, and
  duration (connect logged at debug).
- while an outbound channel stays full, re-warn every 5 minutes with the
  cumulative drop count and how long it has been full; the recovery info line
  reports the full duration.
- doc_id/user on 'Error handling message' and 'Received non-binary message';
  user on the subscription-lock and Subdocs/Event protocol warns.
- the read loop now breaks on stream EOF (as upstream main does) instead of
  waiting on server shutdown with the branch disabled; without this the
  disconnect log would never fire for connections that die without a close
  handshake.
…remaining warns

- info-level 'WebSocket disconnected' with doc_id, user, close_reason, and
  duration (connect logged at debug to keep volume down); makes wedged and
  reconnect-looping clients visible directly in pod logs.
- while an outbound channel stays full, re-warn every 5 minutes with the
  cumulative drop count and how long it has been full; the recovery info line
  now also reports the full duration.
- add doc_id/user to 'Error handling message' and 'Received non-binary
  message', and user to the subscription-lock and Subdocs/Event protocol
  warns in doc_connection.rs.
@petergaultney
petergaultney force-pushed the thds/connection-lifecycle-logs branch from 0bee174 to 21cae26 Compare August 5, 2026 15:09
@petergaultney
petergaultney marked this pull request as ready for review August 5, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant